« Are automated test tools for real? | Main | When To Use a Record-Playback Tool »

February 17, 2005

Automated Testing != Record-Playback Tool

When people hear the word automated testing they conjure up a tool like Mercury's Winrunner, QuickTest Pro or Segue's SilkTest. While these tools seem worthwhile I would argue that for a serious automation effort these tools have a lot of drawbacks.

Maintenance Costs

Record-playback tool vendors like to focus on how easy it is to record a test scenario with their tool. Test writers simply interact with the application normally while the test tool records all their actions. But no matter how easy it is to initially record a test scenario with a record-playback tool, the bulk of time and money will be spent on the ongoing maintenance of each test as the application changes. Screens will get added, buttons will get removed, column names will get modified. A company using a record-playback tool is left with two options - either keep re-recording all the tests as the application changes or stop recording scripts and maintain the generated test scripts like a real program.

Simply re-recording all test scenarios won't hold up in the long term because there will be duplication of recorded test code when testing similar scenarios. When the application under test inevitably changes (e.g. a new button was added which needs to be pressed for each test scenario), test writers will have to re-record all the test scenarios that are affected by that change. Even if the repetitive actions have been factored out into reusable modules, these modules don't use object oriented language features to make them more effective. Record-playback scripts are more expensive to maintain because the code they generate is long, complicated, not object oriented and must be further manipulated to put into reusable components.

It would be more desirable (and economical) to write test code using a real object-oriented language (e.g. Java, C#). Automated web testing design patterns can be fully used to create low maintenance web tests that will withstand continuous changes in the application under test. I have presented and written a paper about automated testing design patterns for the Pacific Northwest Software Quality Conference. These patterns leverage the object-oriented language features to create tests that are low maintenance and adaptable to application changes.

Test Language

Record-playback test tool vendors are not in the business of creating programming languages. Yet most vendors create a language that is specific to their test tool. This forces test developers to learn a specialized language just to test their software. These tools ignore industry-standard programming languages that are meant to be used in a wide array of applications - including testing. There is nothing inherently so strange about test development that necessitates having a unique language just for that task.

By creating their own languages, which are inevitably not as complete as industry standard languages, test developers are left with incomplete libraries for common tasks (e.g. file I/O, string manipulation, collections libraries), which means test developers have to write and test common functions that are already available in standard programming languages. What's more, the languages these vendors come up with are not object-oriented and are designed to keep companies locked-in to their test tools, since no one else uses that language.

This increases the time needed to train test developers on how to use a record-playback tool. Usually, record-playback tool vendors offer training courses to use their programming language with their tool, costing thousands of dollars. Not only are the costs prohibitive, but test developers are less likely to learn a technology if they know it is specific to one vendor's tool.

Design Patterns

Despite what record-playback tool vendors might advertise, test development is a software development activity. As such, the same principles that apply to application development apply to test development. Namely, well designed code that is easy to change, with no duplication. Just as there are well established design patterns to write effective application software, there are analogous test design patterns that enable writing low maintenance test code. Our test tool is well suited to these test design patterns because tests are written in a real object-oriented language instead of a procedural, vendor-specific language.

By using the automated testing design patterns with an object-oriented language, test developers can write tests that insulate them from application changes as well as the test tool. These patterns help decouple vendor-specific test code from application-specific test code, making tests more maintainable and adaptable to change.

Integration with Build and Reporting Processes

Record and playback tools are designed to keep the customer fully dependent on them for all aspects of testing - test creation, execution, reporting. Most use vendor-specific language that are not object-oriented, not recognized by test developers and are not amenable to automated testing design patterns. Even the tools that claim they can generate test code in an object-oriented language generate complicated procedural code that happens to be written in an object-oriented language. This leads to higher test maintenance costs because test developers can't use proven design patterns to lower maintenance costs of tests.

Record and playback tools often sell a total solution, incorporating many products into one test suite. The functional testing component usually uses its own testing framework for test execution, test creation, test reporting, and other test related features. Not only does this lead to serious vendor lock-in when it come to testing your application, but this doesn't address the need of test developers who like to use industry-standard approaches to testing. Most test developers are already familiar with unit testing frameworks such as JUnit. However, they will be forced to use the vendor's testing framework because the vendor's test tool only works with their own framework. If a company ever wanted to switch test tools, it would have to start the process all over again because a record and playback tool is vendor specific in every way - the test code it generates, the way it runs tests, the way it reports test results. This doesn't make it easy to incorporate these tools with industry standard build tools such as Ant.

Posted by Misha Rybalov at February 17, 2005 11:35 AM


Hi All,

It is possible to use with out GUI ( Win runner )
for testing the application ?

Please explain

Thanks & BEst Regards.
Zahir Hussain

Posted by: zahir hussain at July 25, 2006 06:13 AM

Hi, Zahir. Yes, it is possible to use a test tool without a GUI. The one we've developed (www.abcwebtest.com) makes it possible to program the tests in .NET or Java. You program the tests to do what you want them to do, compile the test files and run them like a regular program.

Posted by: Misha Rybalov at August 17, 2006 01:52 PM

Post a comment

Remember Me?