« Job Descriptions as Cubby Holes | Main | Automated Unit Test Creation - Agitar »
February 07, 2005
Automated Testing is Very Manual
I was reading about Autonomic Computing recently where computer systems can automate their own configuration and recover automatically from problems, like our bodies. This made me wonder how this concept can be applied to automated testing. Automated testing is conventionally thought of as automated test execution (i.e. the automatic running of tests). However, there are additional factors that remain a mostly manual process. I will expand on these factors below.
Test Creation
People are still used to creating tests manually. With the tools that I develop and use, test writers must write programming code to tell the tool what the test should do. Similarly, record-play back tools make the test writer record their actions and then later plays them back (aside: this is not way I support developing automated tests for a multitude of reasons - see future blogs). To get to the next generation of automated testing, test developers will need an automated way to create (or at least drastically reduce the time to create) test cases.
Analyzing Test Failures
The accepted way of analyzing test failures is to examine the test results log and determine, based on the discrepancy between the expected and actual values whether the test failure is a bug or a change in the application that requires an auto test update. If there are many test failures, this process can be very time consuming. Again, some form of automation in this area could greatly improve the automated testing process. Perhaps, the automated test tool could be trained (e.g. with a rule engine) to make decisions about whether a test failure is a bug or simply a cosmetic change to the application that warrants an update to the test.
Updating Tests
A reality of automated testing is that the applications that are being tested will inevitably change. When these changes occur, automated tests have to be updated accordingly. To minimize the impact of these changes, auto test developers can make use of design patterns that lower test case maintenance costs. However, one must manually create these design patterns specifically for each application or module under test. I fully advocate this approach but the problem still remains that you have to manually create and then maintain these design patterns. A more mature approach would have the automated test tool automatically create design patterns for an application and then update that code automatically when the application changes.
Entering Bug Reports
Once again, this process is currently very manual. A company will often use a totally different system to enter bugs as it does to run the automated tests (this is the case at my company). So we have this great automated way to run our tests, but then have to rely on totally manual process to enter all the details about the test failure. With a more mature test automation process, the automated test tool could (after determining a test failure is a bug) automatically enter defect reports in the bug tracking system.
Data Setup
Automated testing involves verifying that the actual behaviour of the application matches the expected behaviour. Often, this type of behaviour verification involves checking that expected data matches actual data returned by the application. In order to configure the application to return the expected data, the data has to be pre-loaded into the application (usually into the application's database) before the automated tests are run. This pre-loading, or data setup, falls outside the scope of most automated test tools. Instead, the automated test developer is expected to manage this on their own. This leads to each company having to re-invent its own way of pre-loading its automated test data into its application.
As I see it, automated testing is really in its infancy where tools are mostly focused on automating the test execution side of the testing process and don't adequately address the other aspects to testing such as test creation, test reporting, test updating, entering defects and data setup. With the introduction of these concepts in automated test tools, we can begin to apply principles of autonomic computing in the field of automated testing.
Posted by Misha Rybalov at February 7, 2005 05:23 PM