The details aren't important but if you must know (and there is a selfish side to this post to solicit help) I get a System.InvalidCastException when I try and obtain the "parentWindow" property from an MSHTML.IHTMLDocument2 object. Try as I might to work my way around it, I can't find a way to avoid this exception today.

The really frustrating thing is that this was working before! In previous days, there was no exception. I tried a few techniques to get past this blocker - I took a brief nap to come back with fresh eyes (no luck), I tried researching for a solution to this problem on the web (no luck), I tried reading other people's unrelated blogs to take a break from banging my head against the wall (no luck). So in the end, I decided it's better to stop banging my head against the wall, take my losses for the day and move on to other business activities.

I'm currently exploring the option of using IT recruitment agencies to generate leads. The jury is still out but one thing is for sure - I'm not the run-of-the-mill person they're accustomed to dealing with. For one, I have an automated tested company. For another, I offer an automated web testing tool, (ABCWebTest), that is different from the standard commercial test tools out there. And finally, I have many years of experience implementing low-maintenance automated testing solutions, and not just using record-playback tools (which break down for complex testing scenarios).

While this all presents a nice offering, I find that IT recruiters don't quite know what to do with me. I don't blame them. After all, the companies that hired them are asking for a normal tester to do automated testing. They might not be aware that automated testing takes a lot of planning, technical knowledge and experience to implement properly.

So if you're an IT recruitment company reading this (or a company that's hired a recruitment company for an automated testing consultant), keep in mind - automated testing is a development activity requiring as much forethought, planning, and technical knowledge as regular development. Don't fall for the hype of commercial test tool vendors.

1. Abstract the navigation logic of your application.

There's no need for test cases to know the details of how to navigate around the application. All this does is tightly couple the navigation logic to each test so if the navigation logic changes there could be hundreds of tests to update to use the new navigation. The Transporter pattern deals with this issue.

2. Abstract the test steps

Tests shouldn't have to specify all the test steps explicitly. That makes them tightly coupled to that sequence of test steps, so if a step has to be added or removed that would involve a significant maintenance cost. One solution is to abstract the test steps out to a Template class and make each test implement just a few template methods.

3. Abstract the test data

This is a huge topic that I'm not going to cover thoroughly here. Test data and the management of test data is an acquired skill. Too often, test tools focus on driving the GUI and test data management is an afterthought. And don't tell me that the tool can import your test data from an Excel or CSV file. It's not that simple. What about the scenarios where your tests need the application to be in a certain state before they're run? That would involve direct database access that most test tools lack.

There is also the issue of test data generation. Most tools make you manually specify the test data to use. This is not very automated and leads to a tight coupling between database schema and tests. Instead, there should be a test data layer that will supply tests with data they need. That way tests don't have to hard code data.

4. Abstract the visual details of a page

Tests shouldn't have to know the names of specific widgets on a page they are interacting with. Otherwise, they become tightly coupled to the visual representation of the application and sensitive to any changes in the application. The Domain Test Object (DTO) pattern addresses this problem by modeling the page as an object.

Preparing a JUnit test class

1. create a blank Java file (e.g. FooTest.java)

2. import the following classes:
junit.framework.Test
junit.framework.TestCase
junit.framework.TestSuite

3. create a class that extends from TestCase (e.g. public class FooTest extends TestCase)

4. create a public constructor that takes a string argument called name

e.g. public FooTest(String name) { super(name); }

5. create a public static suite() method that tells JUnit which test cases to run. The easy way to do this is to pass the class instance of your class and let JUnit dynamically figure out which tests to run. It does this by reflecting on your test class (FooTest.class) and looking for all public methods that return void and start with "test".

e.g. public static Test suite() { return new TestSuite(FooTest.class); }

This is all that's required to setup your class for JUnit tests. Now you're ready to actually write some tests.

Writing a JUnit test

1. For every test you'd like to run, create a public method that returns void, takes no arguments and starts with "test"
e.g. public void testBar() { ... }

2. Instantiate the object you'd like to test inside the test method from the previous step.

Foo foo = new Foo();

3. Invoke the method(s) you'd like to test and assign the return values to local variables.

e.g. int actualValue = foo.bar(1, 2, 3);

4. Check the method's return value by using one of JUnit's assert methods. You'll be using methods inside junit.framework.TestCase since you inherited from it. You can get a list of all the assert methods from the JUnit JavaDocs.

e.g. assertEquals(4, actualValue);

Initializing and De-initializing Tests

Often, you'll need to initialize or de-initialize a group of tests in the same way. For example if all your tests had to start with the same object creation, you can place that common initialization code into a special JUnit method called setUp(). Similarly, if all your tests had to do the same de-initializing steps, that code can be placed into a tearDown() method.

Your test methods would then not have to perform those common steps anymore and could start with the "meat" of their testing.

These methods are protected and take no arguments, so if you don't have any use for them, don't include them and by default they will do nothing.

e.g.
//foo variable should be declared as an instance of the test class
protected void setUp() { foo = new Foo(); }
protected void tearDown() { foo.cleanUp(); }

Rules of Thumb

These are general guidelines to keep in mind as you're creating more tests with JUnit.

1. Create a single test class for each application class you have.

2. Name each test class the same name as the application class its testing but with "Test" appended to the end. So if you have an application class called "ShoppingCart.java" the test class would be called "ShoppingCartTest.java"

3. Give meaningful names to each test method. I usually name the test method the same name as the method that its testing with some additional info appended to the name. For instance if I'm testing a method called "add" in my ShoppingCart class, I'll create a few test methods to test different ways of adding to a shopping cart:

• public void testAdd_SingleProduct() { ... }
• public void testAdd_MultipleProducts() { ... }
• public void testAdd_InvalidProduct() { ... }
• public void testAdd_Null() { ... }

4. The scope of how much checking to do in a single test case (test method) is a judgement call. I usually stick to a single method call with a single set of arguments, but you can stuff a single test case with multiple method calls and assertion checks. In the example above, that would mean creating a single testAdd() method and testing adding a single product, multiple products, invalid product and null all in one test method.

Maintenance Costs

Record-playback tool vendors like to focus on how easy it is to record a test scenario with their tool. Test writers simply interact with the application normally while the test tool records all their actions. But no matter how easy it is to initially record a test scenario with a record-playback tool, the bulk of time and money will be spent on the ongoing maintenance of each test as the application changes. Screens will get added, buttons will get removed, column names will get modified. A company using a record-playback tool is left with two options - either keep re-recording all the tests as the application changes or stop recording scripts and maintain the generated test scripts like a real program.

Simply re-recording all test scenarios won't hold up in the long term because there will be duplication of recorded test code when testing similar scenarios. When the application under test inevitably changes (e.g. a new button was added which needs to be pressed for each test scenario), test writers will have to re-record all the test scenarios that are affected by that change. Even if the repetitive actions have been factored out into reusable modules, these modules don't use object oriented language features to make them more effective. Record-playback scripts are more expensive to maintain because the code they generate is long, complicated, not object oriented and must be further manipulated to put into reusable components.

It would be more desirable (and economical) to write test code using a real object-oriented language (e.g. Java, C#). Automated web testing design patterns can be fully used to create low maintenance web tests that will withstand continuous changes in the application under test. I have presented and written a paper about automated testing design patterns for the Pacific Northwest Software Quality Conference. These patterns leverage the object-oriented language features to create tests that are low maintenance and adaptable to application changes.

Test Language

Record-playback test tool vendors are not in the business of creating programming languages. Yet most vendors create a language that is specific to their test tool. This forces test developers to learn a specialized language just to test their software. These tools ignore industry-standard programming languages that are meant to be used in a wide array of applications - including testing. There is nothing inherently so strange about test development that necessitates having a unique language just for that task.

By creating their own languages, which are inevitably not as complete as industry standard languages, test developers are left with incomplete libraries for common tasks (e.g. file I/O, string manipulation, collections libraries), which means test developers have to write and test common functions that are already available in standard programming languages. What's more, the languages these vendors come up with are not object-oriented and are designed to keep companies locked-in to their test tools, since no one else uses that language.

This increases the time needed to train test developers on how to use a record-playback tool. Usually, record-playback tool vendors offer training courses to use their programming language with their tool, costing thousands of dollars. Not only are the costs prohibitive, but test developers are less likely to learn a technology if they know it is specific to one vendor's tool.

Design Patterns

Despite what record-playback tool vendors might advertise, test development is a software development activity. As such, the same principles that apply to application development apply to test development. Namely, well designed code that is easy to change, with no duplication. Just as there are well established design patterns to write effective application software, there are analogous test design patterns that enable writing low maintenance test code. Our test tool is well suited to these test design patterns because tests are written in a real object-oriented language instead of a procedural, vendor-specific language.

By using the automated testing design patterns with an object-oriented language, test developers can write tests that insulate them from application changes as well as the test tool. These patterns help decouple vendor-specific test code from application-specific test code, making tests more maintainable and adaptable to change.

Integration with Build and Reporting Processes

Record and playback tools are designed to keep the customer fully dependent on them for all aspects of testing - test creation, execution, reporting. Most use vendor-specific language that are not object-oriented, not recognized by test developers and are not amenable to automated testing design patterns. Even the tools that claim they can generate test code in an object-oriented language generate complicated procedural code that happens to be written in an object-oriented language. This leads to higher test maintenance costs because test developers can't use proven design patterns to lower maintenance costs of tests.

Record and playback tools often sell a total solution, incorporating many products into one test suite. The functional testing component usually uses its own testing framework for test execution, test creation, test reporting, and other test related features. Not only does this lead to serious vendor lock-in when it come to testing your application, but this doesn't address the need of test developers who like to use industry-standard approaches to testing. Most test developers are already familiar with unit testing frameworks such as JUnit. However, they will be forced to use the vendor's testing framework because the vendor's test tool only works with their own framework. If a company ever wanted to switch test tools, it would have to start the process all over again because a record and playback tool is vendor specific in every way - the test code it generates, the way it runs tests, the way it reports test results. This doesn't make it easy to incorporate these tools with industry standard build tools such as Ant.

He makes the following points:

• even if tools look impressive in a demo, they often fail to deliver on their promise and become shelfware
• the tool is not going to solve all your quality problems for you if there is no testing process nor the right people already in place

• they undermine collaboration between development and testing teams since developers use real programming languages while the test team is left with vendorscript
• vendorscript often lack basic library functions that are present in real programming languages
• no third party books/documentation is available for the vendorscript
• it's hard to find experienced people that are proficient in the vendorscript

It's interesting that a lot of people heard of Agitar while AppPerfect remains relatively unknown.

It would be extremely helpful if this tool could generate tests with input data particular to an application. For instance, if I were using this tool on eBay's code, I'd want it to automatically create test auctions, test bidders, test sellers and so on. That way, you'd be exercising the application that's specific to the rules and objects of that application. In order to make this happen, the tool needs you, the developer, to write a slew of factories that create this application specific data.

It's still unclear to me whether the time taken to create all these factories is less than the time it would take to write out all the unit tests manually. If it is, then this tool has some real value. What's ironic is that at the end of the demo, their sales guy predicted people's skepticism by saying "you probably think this tool is not going to work on your program because you think your program is unique". I thought that was an accurate assessment of the situation. So he's going to come back in a couple of weeks time and redemo this tool against the company's program. Stay tuned...

I find this area fascinating since I feel this could be the next big thing to affect software quality. Automated test creation is one of those aspects of automated testing that has always remained manual (see my previous posts: Automated Testing is Very Manual) and I admire this company for attempting to bring this area to the same level of automation as test execution.

Test Creation

People are still used to creating tests manually. With the tools that I develop and use, test writers must write programming code to tell the tool what the test should do. Similarly, record-play back tools make the test writer record their actions and then later plays them back (aside: this is not way I support developing automated tests for a multitude of reasons - see future blogs). To get to the next generation of automated testing, test developers will need an automated way to create (or at least drastically reduce the time to create) test cases.

Analyzing Test Failures

The accepted way of analyzing test failures is to examine the test results log and determine, based on the discrepancy between the expected and actual values whether the test failure is a bug or a change in the application that requires an auto test update. If there are many test failures, this process can be very time consuming. Again, some form of automation in this area could greatly improve the automated testing process. Perhaps, the automated test tool could be trained (e.g. with a rule engine) to make decisions about whether a test failure is a bug or simply a cosmetic change to the application that warrants an update to the test.

Updating Tests

A reality of automated testing is that the applications that are being tested will inevitably change. When these changes occur, automated tests have to be updated accordingly. To minimize the impact of these changes, auto test developers can make use of design patterns that lower test case maintenance costs. However, one must manually create these design patterns specifically for each application or module under test. I fully advocate this approach but the problem still remains that you have to manually create and then maintain these design patterns. A more mature approach would have the automated test tool automatically create design patterns for an application and then update that code automatically when the application changes.

Entering Bug Reports

Once again, this process is currently very manual. A company will often use a totally different system to enter bugs as it does to run the automated tests (this is the case at my company). So we have this great automated way to run our tests, but then have to rely on totally manual process to enter all the details about the test failure. With a more mature test automation process, the automated test tool could (after determining a test failure is a bug) automatically enter defect reports in the bug tracking system.

Data Setup

Automated testing involves verifying that the actual behaviour of the application matches the expected behaviour. Often, this type of behaviour verification involves checking that expected data matches actual data returned by the application. In order to configure the application to return the expected data, the data has to be pre-loaded into the application (usually into the application's database) before the automated tests are run. This pre-loading, or data setup, falls outside the scope of most automated test tools. Instead, the automated test developer is expected to manage this on their own. This leads to each company having to re-invent its own way of pre-loading its automated test data into its application.

As I see it, automated testing is really in its infancy where tools are mostly focused on automating the test execution side of the testing process and don't adequately address the other aspects to testing such as test creation, test reporting, test updating, entering defects and data setup. With the introduction of these concepts in automated test tools, we can begin to apply principles of autonomic computing in the field of automated testing.