May 05, 2006

NUnit "Quick and Dirty" Tutorial

In the style of the JUnit Quick and Dirty Tutorial, here's a similar tutorial for NUnit. I'm assuming that you're familiar with C#/.NET, downloaded NUnit and have a development environment all setup for compiling and running C# code.

Preparing an NUnit test class

1. create a blank C# file (e.g. FooTests.cs)

2. import the following namespace:
using NUnit.Framework;

3. create a stand alone class that uses the "TestFixture" attribute (e.g. [TestFixture] public class FooTest { })

This is all that's required to setup your class for NUnit tests - fewer steps than for JUnit, in fact. Now you're ready to actually write some tests.


Writing an NUnit test

1. For every test you'd like to run, create a public method that returns void, takes no arguments and uses the "Test" attribute e.g. [Test] public void MyTest() { ... }
Unlike JUnit, there is no need to have the method start with "test" because the "Test" attribute tags the test so that NUnit knows that's your test case.

2. Instantiate the object you'd like to test inside the test method from the previous step.

Foo foo = new Foo();

3. Invoke the method(s) you'd like to test and assign the return values to local variables.

e.g. int actualValue = foo.bar(1, 2, 3);

4. Check the method's return value by using one of NUnit's assert methods. You can get a list of all the assert methods from the NUnit Documentation.

e.g. Assert.AreEqual(4, actualValue);


Initializing and De-initializing Tests

Often, you'll need to initialize or de-initialize a group of tests in the same way. For example if all your tests had to start with the same object creation, you can place that common initialization code into a special NUnit method with the "SetUp" attribute. Similarly, if all your tests had to do the same de-initializing steps, that code can be placed into a method tagged with the "TearDown" attribute.

Your test methods would then not have to perform those common steps anymore and could start with the "meat" of their testing.

The set up and tear down methods can have any name (unlike JUnit which requires they be called setUp() and tearDown(), respectively. Also, these methods don't need to be protected like in JUnit but can be public. If you don't have any use for them, don't include them and by default there will be no set up and tear down.

e.g.
//foo variable should be declared as an instance of the test class
[SetUp] public void SetUp() { foo = new Foo(); }
[TearDown] public void TearDown() { foo.cleanUp(); }

NUnit also has a nice once-only initialization/de-initialization feature where you can do an initialization once before any tests are run (instead of once before every test is run) and once after all tests are run (instead of once after every test is run). Use the "TestFixtureSetUp" and "TestFixtureTearDown" attributes, respectively, like so:

[TestFixtureSetUp] public void TestFixtureSetUp() {
//...
}

[TestFixtureTearDown] public void TestFixtureTearDown() {
//...
}

Grouping Your Tests Into a Suite
You can have a master suite test file that is a collection of all your unit test classes. This is another stand alone C# class that's tagged with the "Suite" attribute. Here's a sample of how it would look like:


using NUnit.Framework;
using NUnit.Core;

namespace ABCWebTest.Tests {
   public class AllTests {
      [Suite] public static TestSuite Suite {
         get {
            TestSuite suite = new TestSuite("All Tests");
            suite.Add(new AjaxTests());
            suite.Add(new BrowserTests());
            suite.Add(new BrowserUtilityTests());
            suite.Add(new KeyboardTests());
            suite.Add(new MouseTests());
            suite.Add(new PopupTests());
            suite.Add(new SpeedTests());
            suite.Add(new TextTests());
            suite.Add(new TimerTests());
            return suite;
         }
      }
   }
}


Rules of Thumb

These are general guidelines to keep in mind as you're creating more tests with NUnit.

1. Create a single test class for each application class you have.

2. Name each test class the same name as the application class its testing but with "Tests" appended to the end. So if you have an application class called "ShoppingCart.cs" the test class would be called "ShoppingCartTests.cs"

3. Give meaningful names to each test method. I usually name the test method the same name as the method that its testing with some additional info appended to the name. For instance if I'm testing a method called "add" in my ShoppingCart class, I'll create a few test methods to test different ways of adding to a shopping cart:

4. The scope of how much checking to do in a single test case (test method) is a judgement call. I usually stick to a single method call with a single set of arguments, but you can stuff a single test case with multiple method calls and assertion checks. In the example above, that would mean creating a single Add() method and testing adding a single product, multiple products, invalid product and null all in one test method.

5. You can use the NUnit console or NUnit GUI to run your tests. The console I find is quicker. I usually use NAnt to run my NUnit tests from the console but I also find it convenient to use the NUnit GUI when I want to run a subset of my tests. Keep in mind, though, that the NUnit GUI uses the GUI thread as the main thread so tests aren't run in the main thread. This may cause issues if your code expects to be running in the main thread - I learned this lesson the hard way.

Posted by Misha Rybalov at 06:00 PM | Comments (0)

March 05, 2005

JUnit "Quick and Dirty" Tutorial

For those that want to get started with JUnit quickly and prefer a "just the facts" type of tutorial, I've create the following JUnit Quick and Dirty Tutorial. This tutorial assumes you are already familiar with Java programming, so the junit.jar should be in your classpath and you should have a development environment all setup for compiling and running Java code.

Preparing a JUnit test class

1. create a blank Java file (e.g. FooTest.java)

2. import the following classes:
junit.framework.Test
junit.framework.TestCase
junit.framework.TestSuite

3. create a class that extends from TestCase (e.g. public class FooTest extends TestCase)

4. create a public constructor that takes a string argument called name

e.g. public FooTest(String name) { super(name); }

5. create a public static suite() method that tells JUnit which test cases to run. The easy way to do this is to pass the class instance of your class and let JUnit dynamically figure out which tests to run. It does this by reflecting on your test class (FooTest.class) and looking for all public methods that return void and start with "test".

e.g. public static Test suite() { return new TestSuite(FooTest.class); }

This is all that's required to setup your class for JUnit tests. Now you're ready to actually write some tests.

Writing a JUnit test

1. For every test you'd like to run, create a public method that returns void, takes no arguments and starts with "test"
e.g. public void testBar() { ... }

2. Instantiate the object you'd like to test inside the test method from the previous step.

Foo foo = new Foo();

3. Invoke the method(s) you'd like to test and assign the return values to local variables.

e.g. int actualValue = foo.bar(1, 2, 3);

4. Check the method's return value by using one of JUnit's assert methods. You'll be using methods inside junit.framework.TestCase since you inherited from it. You can get a list of all the assert methods from the JUnit JavaDocs.

e.g. assertEquals(4, actualValue);

Initializing and De-initializing Tests

Often, you'll need to initialize or de-initialize a group of tests in the same way. For example if all your tests had to start with the same object creation, you can place that common initialization code into a special JUnit method called setUp(). Similarly, if all your tests had to do the same de-initializing steps, that code can be placed into a tearDown() method.

Your test methods would then not have to perform those common steps anymore and could start with the "meat" of their testing.

These methods are protected and take no arguments, so if you don't have any use for them, don't include them and by default they will do nothing.

e.g.
//foo variable should be declared as an instance of the test class
protected void setUp() { foo = new Foo(); }
protected void tearDown() { foo.cleanUp(); }

Rules of Thumb

These are general guidelines to keep in mind as you're creating more tests with JUnit.

1. Create a single test class for each application class you have.

2. Name each test class the same name as the application class its testing but with "Test" appended to the end. So if you have an application class called "ShoppingCart.java" the test class would be called "ShoppingCartTest.java"

3. Give meaningful names to each test method. I usually name the test method the same name as the method that its testing with some additional info appended to the name. For instance if I'm testing a method called "add" in my ShoppingCart class, I'll create a few test methods to test different ways of adding to a shopping cart:

4. The scope of how much checking to do in a single test case (test method) is a judgement call. I usually stick to a single method call with a single set of arguments, but you can stuff a single test case with multiple method calls and assertion checks. In the example above, that would mean creating a single testAdd() method and testing adding a single product, multiple products, invalid product and null all in one test method.

Posted by Misha Rybalov at 12:15 PM | Comments (3)

February 18, 2005

When To Use a Record-Playback Tool

As I previously stated, record-playback tools have some serious limitations. But that doesn't mean they should never be used. Here are some cases where they can add value:

  • A completely non-technical person/department is responsible for automating tests

    • These tools boast about their ease of use and how quickly a non-technical person can generate an automated test script. The person can just record a scenario and then play it back automatically.

  • The application changes relatively infrequently

    • Unless the automation strategy involves some use of design patterns it's going to be very difficult to maintain new automated tests. What's most likely to happen is the automated testing resource will need to re-record test scenarios whenever the application changes. As long as this happens relatively infrequently, it could be a manageable cost.

    Posted by Misha Rybalov at 12:08 PM | Comments (0)

    February 17, 2005

    Automated Testing != Record-Playback Tool

    When people hear the word automated testing they conjure up a tool like Mercury's Winrunner, QuickTest Pro or Segue's SilkTest. While these tools seem worthwhile I would argue that for a serious automation effort these tools have a lot of drawbacks.

    Maintenance Costs

    Record-playback tool vendors like to focus on how easy it is to record a test scenario with their tool. Test writers simply interact with the application normally while the test tool records all their actions. But no matter how easy it is to initially record a test scenario with a record-playback tool, the bulk of time and money will be spent on the ongoing maintenance of each test as the application changes. Screens will get added, buttons will get removed, column names will get modified. A company using a record-playback tool is left with two options - either keep re-recording all the tests as the application changes or stop recording scripts and maintain the generated test scripts like a real program.

    Simply re-recording all test scenarios won't hold up in the long term because there will be duplication of recorded test code when testing similar scenarios. When the application under test inevitably changes (e.g. a new button was added which needs to be pressed for each test scenario), test writers will have to re-record all the test scenarios that are affected by that change. Even if the repetitive actions have been factored out into reusable modules, these modules don't use object oriented language features to make them more effective. Record-playback scripts are more expensive to maintain because the code they generate is long, complicated, not object oriented and must be further manipulated to put into reusable components.

    It would be more desirable (and economical) to write test code using a real object-oriented language (e.g. Java, C#). Automated web testing design patterns can be fully used to create low maintenance web tests that will withstand continuous changes in the application under test. I have presented and written a paper about automated testing design patterns for the Pacific Northwest Software Quality Conference. These patterns leverage the object-oriented language features to create tests that are low maintenance and adaptable to application changes.

    Test Language

    Record-playback test tool vendors are not in the business of creating programming languages. Yet most vendors create a language that is specific to their test tool. This forces test developers to learn a specialized language just to test their software. These tools ignore industry-standard programming languages that are meant to be used in a wide array of applications - including testing. There is nothing inherently so strange about test development that necessitates having a unique language just for that task.

    By creating their own languages, which are inevitably not as complete as industry standard languages, test developers are left with incomplete libraries for common tasks (e.g. file I/O, string manipulation, collections libraries), which means test developers have to write and test common functions that are already available in standard programming languages. What's more, the languages these vendors come up with are not object-oriented and are designed to keep companies locked-in to their test tools, since no one else uses that language.

    This increases the time needed to train test developers on how to use a record-playback tool. Usually, record-playback tool vendors offer training courses to use their programming language with their tool, costing thousands of dollars. Not only are the costs prohibitive, but test developers are less likely to learn a technology if they know it is specific to one vendor's tool.


    Design Patterns

    Despite what record-playback tool vendors might advertise, test development is a software development activity. As such, the same principles that apply to application development apply to test development. Namely, well designed code that is easy to change, with no duplication. Just as there are well established design patterns to write effective application software, there are analogous test design patterns that enable writing low maintenance test code. Our test tool is well suited to these test design patterns because tests are written in a real object-oriented language instead of a procedural, vendor-specific language.

    By using the automated testing design patterns with an object-oriented language, test developers can write tests that insulate them from application changes as well as the test tool. These patterns help decouple vendor-specific test code from application-specific test code, making tests more maintainable and adaptable to change.


    Integration with Build and Reporting Processes

    Record and playback tools are designed to keep the customer fully dependent on them for all aspects of testing - test creation, execution, reporting. Most use vendor-specific language that are not object-oriented, not recognized by test developers and are not amenable to automated testing design patterns. Even the tools that claim they can generate test code in an object-oriented language generate complicated procedural code that happens to be written in an object-oriented language. This leads to higher test maintenance costs because test developers can't use proven design patterns to lower maintenance costs of tests.

    Record and playback tools often sell a total solution, incorporating many products into one test suite. The functional testing component usually uses its own testing framework for test execution, test creation, test reporting, and other test related features. Not only does this lead to serious vendor lock-in when it come to testing your application, but this doesn't address the need of test developers who like to use industry-standard approaches to testing. Most test developers are already familiar with unit testing frameworks such as JUnit. However, they will be forced to use the vendor's testing framework because the vendor's test tool only works with their own framework. If a company ever wanted to switch test tools, it would have to start the process all over again because a record and playback tool is vendor specific in every way - the test code it generates, the way it runs tests, the way it reports test results. This doesn't make it easy to incorporate these tools with industry standard build tools such as Ant.

    Posted by Misha Rybalov at 11:35 AM | Comments (2)

    February 16, 2005

    Are automated test tools for real?

    Alan Earls wrote an article comparing pros and cons of traditional automated testing tools.

    He makes the following points:

    Posted by Misha Rybalov at 10:51 AM | Comments (1)

    February 15, 2005

    Hey Vendors, Give Us Real Scripting Languages

    Bret Pettichord wrote an article about how traditional automated testing tools force their users to use a proprietary language to automate their tests. He makes a lot of great points about the downside of using these proprietary languages (vendorscript) for test automation:

    Posted by Misha Rybalov at 05:15 PM | Comments (0)

    February 14, 2005

    Automated Unit Test Creation - AppPerfect

    After the demo with Agitar, a fellow developer discovered another product that does what Agitator does and then some for a fraction of Agitar's cost. AppPerfect includes automated unit test creation, code analyzer, a web testing tool (integrates with HttpUnit) and a load testing tool. They have a free version of the tool that only omits a few features and a paid version which is priced at $495 per seat - about 1/7th the cost of Agitar's per seat cost.

    It's interesting that a lot of people heard of Agitar while AppPerfect remains relatively unknown.

    Posted by Misha Rybalov at 04:59 PM | Comments (0)

    February 11, 2005

    Automated Unit Test Creation - Agitar

    Today at work, we got a demo of a piece of software that automates the creation of unit tests for an arbitrary piece of Java code. After going through the obligatory slides, they got to the meat of the presentation by generating a whole slew of tests for a samples piece of code (they call this process "agitation"). While it was impressive, I'm still not fully convinced that it can do this intelligently with a system it knows nothing about. A lot of the test cases were boundary conditions for methods while others were garbage type inputs to see if the code would break. While this type of input testing is very important it doesn't cover the whole spectrum of tests that need to be generated.

    It would be extremely helpful if this tool could generate tests with input data particular to an application. For instance, if I were using this tool on eBay's code, I'd want it to automatically create test auctions, test bidders, test sellers and so on. That way, you'd be exercising the application that's specific to the rules and objects of that application. In order to make this happen, the tool needs you, the developer, to write a slew of factories that create this application specific data.

    It's still unclear to me whether the time taken to create all these factories is less than the time it would take to write out all the unit tests manually. If it is, then this tool has some real value. What's ironic is that at the end of the demo, their sales guy predicted people's skepticism by saying "you probably think this tool is not going to work on your program because you think your program is unique". I thought that was an accurate assessment of the situation. So he's going to come back in a couple of weeks time and redemo this tool against the company's program. Stay tuned...

    I find this area fascinating since I feel this could be the next big thing to affect software quality. Automated test creation is one of those aspects of automated testing that has always remained manual (see my previous posts: Automated Testing is Very Manual) and I admire this company for attempting to bring this area to the same level of automation as test execution.

    Posted by Misha Rybalov at 05:01 PM | Comments (0)