February 07, 2005
Automated Testing is Very Manual
I was reading about Autonomic Computing recently where computer systems can automate their own configuration and recover automatically from problems, like our bodies. This made me wonder how this concept can be applied to automated testing. Automated testing is conventionally thought of as automated test execution (i.e. the automatic running of tests). However, there are additional factors that remain a mostly manual process. I will expand on these factors below.
Test Creation
People are still used to creating tests manually. With the tools that I develop and use, test writers must write programming code to tell the tool what the test should do. Similarly, record-play back tools make the test writer record their actions and then later plays them back (aside: this is not way I support developing automated tests for a multitude of reasons - see future blogs). To get to the next generation of automated testing, test developers will need an automated way to create (or at least drastically reduce the time to create) test cases.
Analyzing Test Failures
The accepted way of analyzing test failures is to examine the test results log and determine, based on the discrepancy between the expected and actual values whether the test failure is a bug or a change in the application that requires an auto test update. If there are many test failures, this process can be very time consuming. Again, some form of automation in this area could greatly improve the automated testing process. Perhaps, the automated test tool could be trained (e.g. with a rule engine) to make decisions about whether a test failure is a bug or simply a cosmetic change to the application that warrants an update to the test.
Updating Tests
A reality of automated testing is that the applications that are being tested will inevitably change. When these changes occur, automated tests have to be updated accordingly. To minimize the impact of these changes, auto test developers can make use of design patterns that lower test case maintenance costs. However, one must manually create these design patterns specifically for each application or module under test. I fully advocate this approach but the problem still remains that you have to manually create and then maintain these design patterns. A more mature approach would have the automated test tool automatically create design patterns for an application and then update that code automatically when the application changes.
Entering Bug Reports
Once again, this process is currently very manual. A company will often use a totally different system to enter bugs as it does to run the automated tests (this is the case at my company). So we have this great automated way to run our tests, but then have to rely on totally manual process to enter all the details about the test failure. With a more mature test automation process, the automated test tool could (after determining a test failure is a bug) automatically enter defect reports in the bug tracking system.
Data Setup
Automated testing involves verifying that the actual behaviour of the application matches the expected behaviour. Often, this type of behaviour verification involves checking that expected data matches actual data returned by the application. In order to configure the application to return the expected data, the data has to be pre-loaded into the application (usually into the application's database) before the automated tests are run. This pre-loading, or data setup, falls outside the scope of most automated test tools. Instead, the automated test developer is expected to manage this on their own. This leads to each company having to re-invent its own way of pre-loading its automated test data into its application.
As I see it, automated testing is really in its infancy where tools are mostly focused on automating the test execution side of the testing process and don't adequately address the other aspects to testing such as test creation, test reporting, test updating, entering defects and data setup. With the introduction of these concepts in automated test tools, we can begin to apply principles of autonomic computing in the field of automated testing.
Posted by Misha Rybalov at 05:23 PM | Comments (0)
February 05, 2005
Job Descriptions as Cubby Holes
One of my big beefs about standard job descriptions at software companies is that they tend to place people in professional cubby holes. The classic examples are the "developer" and "tester" job descriptions. The developer is supposed to write software while the tester is supposed to test the software. But what if you're a developer that happens to test a lot of their own software? What if you're a developer that believes that testing is just as important as the creation of new software? To me, these job descriptions just perpetuate the myth that developers aren't responsible for testing and instead can pass this responsibility to someone else. After all, that someone else is officially referred to as the "tester" so it's their responsibility to make sure that my code works properly, right? I disagree.
My profession involves developing automated testing systems for companies and help them adopt automated testing into their development process. So I happen to be a developer that likes to develop software that automates the testing of other software. This throws a monkey wrench into the normal HR job title process because I don't fit nicely into the developer or tester cubby hole.
Posted by Misha Rybalov at 05:56 PM | Comments (0)
February 01, 2005
Development = = Testing
In most software development organizations there is a clear division between the people who write software and the people that test the software. I always felt uneasy about this division, especially during the last few years where I've been fortunate enough to work on both sides of this divide. The fundamental assumption in this organizational scheme is that you need different people to do the testing than you need to write the software. I have several concerns with this.
Testing is Part of Software Development
Since when did testing become an outside activity to development? Maybe I just don't get this concept, but to me, testing is fundamental to software development. Take any developer who hates testing their code and I'll show you someone who happens to test their code in a very unstructured or lazy way. It doesn't mean that they don't test their code. Otherwise, how could they have even the slightest confidence in the code they produce? So what most developers end up doing is some minimal, manual testing to make sure a few superficial test cases pass and then declare the code as "complete" and pass it on to the testing department for more thorough testing.
What we, as developers, are essentially doing is passing the buck to someone else to verify the quality of our code.
Are Testers Necessary?
Yes they are. There will always be a need for a group of people to try and break software that developers produce. What I am saying, though, is that testers should be given higher level testing responsibilities. We shouldn't be relying on testers to answer questions like "does the software do the basic things it's supposed to do?". Instead, testers can answer questions like:
Currently, testers are forced to answer just one question: does the software work? From my experience, that's what they spend the majority of their time figuring out. As developers, WE should be answering that question and free up tester's time to answer the more complicated questions about our software. As developers, we should be taking more responsibility for the quality of the software we produce.
Posted by Misha Rybalov at 05:59 PM | Comments (0)
January 20, 2005
Testing Anti Lock Brakes
Sue has recently purchased a new car and it was very important for her that the car have anti-lock brakes (ABS). We went together to test drive the cars and I was determined to test the ABS in each car we drove. It was reassuring to hear the ABS system start pumping as I made a hard stop, but it made me wonder - do all people that get cars with ABS test that feature? Also, isn't it unfair to expect the customer to test that feature? Shouldn't the manufacturer test it out, too? I will fully admit that I know very little about the car manufacturing process, so I'm just curious - how DOES the manufacturer test that each car with ABS has it working correctly? I'm guessing that it doesn't test each individually because that would be infeasible. I'm guessing they test a few "throw away" cars that they crash to make sure all the safety system work properly.
But this made me think about how difficult it can sometimes be to write software that's testable - especially if you don't think about testing from the start.
I'm happy to say that Sue signed the contract yesterday and is the happy owner of a 2005 Toyota Corolla LE. We used a service that gives us the dealer cost for cars and I have to say it was well worth the price. This made the whole negotiating process a lot easier.
Posted by Misha Rybalov at 01:30 PM | Comments (0)