« January 2005 | Main | March 2005 »
February 18, 2005
When To Use a Record-Playback Tool
As I previously stated, record-playback tools have some serious limitations. But that doesn't mean they should never be used. Here are some cases where they can add value:
These tools boast about their ease of use and how quickly a non-technical person can generate an automated test script. The person can just record a scenario and then play it back automatically.
Unless the automation strategy involves some use of design patterns it's going to be very difficult to maintain new automated tests. What's most likely to happen is the automated testing resource will need to re-record test scenarios whenever the application changes. As long as this happens relatively infrequently, it could be a manageable cost.
Posted by Misha Rybalov at 12:08 PM | Comments (0)
February 17, 2005
Automated Testing != Record-Playback Tool
When people hear the word automated testing they conjure up a tool like Mercury's Winrunner, QuickTest Pro or Segue's SilkTest. While these tools seem worthwhile I would argue that for a serious automation effort these tools have a lot of drawbacks.
Maintenance Costs
Record-playback tool vendors like to focus on how easy it is to record a test scenario with their tool. Test writers simply interact with the application normally while the test tool records all their actions. But no matter how easy it is to initially record a test scenario with a record-playback tool, the bulk of time and money will be spent on the ongoing maintenance of each test as the application changes. Screens will get added, buttons will get removed, column names will get modified. A company using a record-playback tool is left with two options - either keep re-recording all the tests as the application changes or stop recording scripts and maintain the generated test scripts like a real program.
Simply re-recording all test scenarios won't hold up in the long term because there will be duplication of recorded test code when testing similar scenarios. When the application under test inevitably changes (e.g. a new button was added which needs to be pressed for each test scenario), test writers will have to re-record all the test scenarios that are affected by that change. Even if the repetitive actions have been factored out into reusable modules, these modules don't use object oriented language features to make them more effective. Record-playback scripts are more expensive to maintain because the code they generate is long, complicated, not object oriented and must be further manipulated to put into reusable components.
It would be more desirable (and economical) to write test code using a real object-oriented language (e.g. Java, C#). Automated web testing design patterns can be fully used to create low maintenance web tests that will withstand continuous changes in the application under test. I have presented and written a paper about automated testing design patterns for the Pacific Northwest Software Quality Conference. These patterns leverage the object-oriented language features to create tests that are low maintenance and adaptable to application changes.
Test Language
Record-playback test tool vendors are not in the business of creating programming languages. Yet most vendors create a language that is specific to their test tool. This forces test developers to learn a specialized language just to test their software. These tools ignore industry-standard programming languages that are meant to be used in a wide array of applications - including testing. There is nothing inherently so strange about test development that necessitates having a unique language just for that task.
By creating their own languages, which are inevitably not as complete as industry standard languages, test developers are left with incomplete libraries for common tasks (e.g. file I/O, string manipulation, collections libraries), which means test developers have to write and test common functions that are already available in standard programming languages. What's more, the languages these vendors come up with are not object-oriented and are designed to keep companies locked-in to their test tools, since no one else uses that language.
This increases the time needed to train test developers on how to use a record-playback tool. Usually, record-playback tool vendors offer training courses to use their programming language with their tool, costing thousands of dollars. Not only are the costs prohibitive, but test developers are less likely to learn a technology if they know it is specific to one vendor's tool.
Design Patterns
Despite what record-playback tool vendors might advertise, test development is a software development activity. As such, the same principles that apply to application development apply to test development. Namely, well designed code that is easy to change, with no duplication. Just as there are well established design patterns to write effective application software, there are analogous test design patterns that enable writing low maintenance test code. Our test tool is well suited to these test design patterns because tests are written in a real object-oriented language instead of a procedural, vendor-specific language.
By using the automated testing design patterns with an object-oriented language, test developers can write tests that insulate them from application changes as well as the test tool. These patterns help decouple vendor-specific test code from application-specific test code, making tests more maintainable and adaptable to change.
Integration with Build and Reporting Processes
Record and playback tools are designed to keep the customer fully dependent on them for all aspects of testing - test creation, execution, reporting. Most use vendor-specific language that are not object-oriented, not recognized by test developers and are not amenable to automated testing design patterns. Even the tools that claim they can generate test code in an object-oriented language generate complicated procedural code that happens to be written in an object-oriented language. This leads to higher test maintenance costs because test developers can't use proven design patterns to lower maintenance costs of tests.
Record and playback tools often sell a total solution, incorporating many products into one test suite. The functional testing component usually uses its own testing framework for test execution, test creation, test reporting, and other test related features. Not only does this lead to serious vendor lock-in when it come to testing your application, but this doesn't address the need of test developers who like to use industry-standard approaches to testing. Most test developers are already familiar with unit testing frameworks such as JUnit. However, they will be forced to use the vendor's testing framework because the vendor's test tool only works with their own framework. If a company ever wanted to switch test tools, it would have to start the process all over again because a record and playback tool is vendor specific in every way - the test code it generates, the way it runs tests, the way it reports test results. This doesn't make it easy to incorporate these tools with industry standard build tools such as Ant.
Posted by Misha Rybalov at 11:35 AM | Comments (2)
February 16, 2005
Are automated test tools for real?
Alan Earls wrote an article comparing pros and cons of traditional automated testing tools.
He makes the following points:
- even if tools look impressive in a demo, they often fail to deliver on their promise and become shelfware
- the tool is not going to solve all your quality problems for you if there is no testing process nor the right people already in place
Posted by Misha Rybalov at 10:51 AM | Comments (1)
February 15, 2005
Hey Vendors, Give Us Real Scripting Languages
Bret Pettichord wrote an article about how traditional automated testing tools force their users to use a proprietary language to automate their tests. He makes a lot of great points about the downside of using these proprietary languages (vendorscript) for test automation:
- they undermine collaboration between development and testing teams since developers use real programming languages while the test team is left with vendorscript
- vendorscript often lack basic library functions that are present in real programming languages
- no third party books/documentation is available for the vendorscript
- it's hard to find experienced people that are proficient in the vendorscript
Posted by Misha Rybalov at 05:15 PM | Comments (0)
February 14, 2005
Automated Unit Test Creation - AppPerfect
After the demo with Agitar, a fellow developer discovered another product that does what Agitator does and then some for a fraction of Agitar's cost. AppPerfect includes automated unit test creation, code analyzer, a web testing tool (integrates with HttpUnit) and a load testing tool. They have a free version of the tool that only omits a few features and a paid version which is priced at $495 per seat - about 1/7th the cost of Agitar's per seat cost.
It's interesting that a lot of people heard of Agitar while AppPerfect remains relatively unknown.
Posted by Misha Rybalov at 04:59 PM | Comments (0)
February 11, 2005
Automated Unit Test Creation - Agitar
Today at work, we got a demo of a piece of software that automates the creation of unit tests for an arbitrary piece of Java code. After going through the obligatory slides, they got to the meat of the presentation by generating a whole slew of tests for a samples piece of code (they call this process "agitation"). While it was impressive, I'm still not fully convinced that it can do this intelligently with a system it knows nothing about. A lot of the test cases were boundary conditions for methods while others were garbage type inputs to see if the code would break. While this type of input testing is very important it doesn't cover the whole spectrum of tests that need to be generated.
It would be extremely helpful if this tool could generate tests with input data particular to an application. For instance, if I were using this tool on eBay's code, I'd want it to automatically create test auctions, test bidders, test sellers and so on. That way, you'd be exercising the application that's specific to the rules and objects of that application. In order to make this happen, the tool needs you, the developer, to write a slew of factories that create this application specific data.
It's still unclear to me whether the time taken to create all these factories is less than the time it would take to write out all the unit tests manually. If it is, then this tool has some real value. What's ironic is that at the end of the demo, their sales guy predicted people's skepticism by saying "you probably think this tool is not going to work on your program because you think your program is unique". I thought that was an accurate assessment of the situation. So he's going to come back in a couple of weeks time and redemo this tool against the company's program. Stay tuned...
I find this area fascinating since I feel this could be the next big thing to affect software quality. Automated test creation is one of those aspects of automated testing that has always remained manual (see my previous posts: Automated Testing is Very Manual) and I admire this company for attempting to bring this area to the same level of automation as test execution.
Posted by Misha Rybalov at 05:01 PM | Comments (0)
February 07, 2005
Automated Testing is Very Manual
I was reading about Autonomic Computing recently where computer systems can automate their own configuration and recover automatically from problems, like our bodies. This made me wonder how this concept can be applied to automated testing. Automated testing is conventionally thought of as automated test execution (i.e. the automatic running of tests). However, there are additional factors that remain a mostly manual process. I will expand on these factors below.
Test Creation
People are still used to creating tests manually. With the tools that I develop and use, test writers must write programming code to tell the tool what the test should do. Similarly, record-play back tools make the test writer record their actions and then later plays them back (aside: this is not way I support developing automated tests for a multitude of reasons - see future blogs). To get to the next generation of automated testing, test developers will need an automated way to create (or at least drastically reduce the time to create) test cases.
Analyzing Test Failures
The accepted way of analyzing test failures is to examine the test results log and determine, based on the discrepancy between the expected and actual values whether the test failure is a bug or a change in the application that requires an auto test update. If there are many test failures, this process can be very time consuming. Again, some form of automation in this area could greatly improve the automated testing process. Perhaps, the automated test tool could be trained (e.g. with a rule engine) to make decisions about whether a test failure is a bug or simply a cosmetic change to the application that warrants an update to the test.
Updating Tests
A reality of automated testing is that the applications that are being tested will inevitably change. When these changes occur, automated tests have to be updated accordingly. To minimize the impact of these changes, auto test developers can make use of design patterns that lower test case maintenance costs. However, one must manually create these design patterns specifically for each application or module under test. I fully advocate this approach but the problem still remains that you have to manually create and then maintain these design patterns. A more mature approach would have the automated test tool automatically create design patterns for an application and then update that code automatically when the application changes.
Entering Bug Reports
Once again, this process is currently very manual. A company will often use a totally different system to enter bugs as it does to run the automated tests (this is the case at my company). So we have this great automated way to run our tests, but then have to rely on totally manual process to enter all the details about the test failure. With a more mature test automation process, the automated test tool could (after determining a test failure is a bug) automatically enter defect reports in the bug tracking system.
Data Setup
Automated testing involves verifying that the actual behaviour of the application matches the expected behaviour. Often, this type of behaviour verification involves checking that expected data matches actual data returned by the application. In order to configure the application to return the expected data, the data has to be pre-loaded into the application (usually into the application's database) before the automated tests are run. This pre-loading, or data setup, falls outside the scope of most automated test tools. Instead, the automated test developer is expected to manage this on their own. This leads to each company having to re-invent its own way of pre-loading its automated test data into its application.
As I see it, automated testing is really in its infancy where tools are mostly focused on automating the test execution side of the testing process and don't adequately address the other aspects to testing such as test creation, test reporting, test updating, entering defects and data setup. With the introduction of these concepts in automated test tools, we can begin to apply principles of autonomic computing in the field of automated testing.
Posted by Misha Rybalov at 05:23 PM | Comments (0)
February 05, 2005
Job Descriptions as Cubby Holes
One of my big beefs about standard job descriptions at software companies is that they tend to place people in professional cubby holes. The classic examples are the "developer" and "tester" job descriptions. The developer is supposed to write software while the tester is supposed to test the software. But what if you're a developer that happens to test a lot of their own software? What if you're a developer that believes that testing is just as important as the creation of new software? To me, these job descriptions just perpetuate the myth that developers aren't responsible for testing and instead can pass this responsibility to someone else. After all, that someone else is officially referred to as the "tester" so it's their responsibility to make sure that my code works properly, right? I disagree.
My profession involves developing automated testing systems for companies and help them adopt automated testing into their development process. So I happen to be a developer that likes to develop software that automates the testing of other software. This throws a monkey wrench into the normal HR job title process because I don't fit nicely into the developer or tester cubby hole.
Posted by Misha Rybalov at 05:56 PM | Comments (0)
February 01, 2005
Development = = Testing
In most software development organizations there is a clear division between the people who write software and the people that test the software. I always felt uneasy about this division, especially during the last few years where I've been fortunate enough to work on both sides of this divide. The fundamental assumption in this organizational scheme is that you need different people to do the testing than you need to write the software. I have several concerns with this.
Testing is Part of Software Development
Since when did testing become an outside activity to development? Maybe I just don't get this concept, but to me, testing is fundamental to software development. Take any developer who hates testing their code and I'll show you someone who happens to test their code in a very unstructured or lazy way. It doesn't mean that they don't test their code. Otherwise, how could they have even the slightest confidence in the code they produce? So what most developers end up doing is some minimal, manual testing to make sure a few superficial test cases pass and then declare the code as "complete" and pass it on to the testing department for more thorough testing.
What we, as developers, are essentially doing is passing the buck to someone else to verify the quality of our code.
Are Testers Necessary?
Yes they are. There will always be a need for a group of people to try and break software that developers produce. What I am saying, though, is that testers should be given higher level testing responsibilities. We shouldn't be relying on testers to answer questions like "does the software do the basic things it's supposed to do?". Instead, testers can answer questions like:
Currently, testers are forced to answer just one question: does the software work? From my experience, that's what they spend the majority of their time figuring out. As developers, WE should be answering that question and free up tester's time to answer the more complicated questions about our software. As developers, we should be taking more responsibility for the quality of the software we produce.
Posted by Misha Rybalov at 05:59 PM | Comments (0)