« December 2004 | Main | February 2005 »

January 20, 2005

Testing Anti Lock Brakes

Sue has recently purchased a new car and it was very important for her that the car have anti-lock brakes (ABS). We went together to test drive the cars and I was determined to test the ABS in each car we drove. It was reassuring to hear the ABS system start pumping as I made a hard stop, but it made me wonder - do all people that get cars with ABS test that feature? Also, isn't it unfair to expect the customer to test that feature? Shouldn't the manufacturer test it out, too? I will fully admit that I know very little about the car manufacturing process, so I'm just curious - how DOES the manufacturer test that each car with ABS has it working correctly? I'm guessing that it doesn't test each individually because that would be infeasible. I'm guessing they test a few "throw away" cars that they crash to make sure all the safety system work properly.
But this made me think about how difficult it can sometimes be to write software that's testable - especially if you don't think about testing from the start.

I'm happy to say that Sue signed the contract yesterday and is the happy owner of a 2005 Toyota Corolla LE. We used a service that gives us the dealer cost for cars and I have to say it was well worth the price. This made the whole negotiating process a lot easier.

Posted by Misha Rybalov at 01:30 PM | Comments (0)

January 15, 2005

0. Design Patterns for Customer Testing

This series of posts is based on a paper I wrote and presented at the Pacific Northwest Software Quality Conference in October 2004. If you prefer to read the paper in pdf format you can download it here.

Posted by Misha Rybalov at 07:28 AM | Comments (4)

January 10, 2005

1. Abstract

This paper presents design patterns for automated customer tests. As applications continue to evolve, these patterns help lower maintenance costs and facilitate the creation of new customer tests. Each design pattern includes a definition, a discussion of the pattern details highlighting its suitability for customer test automation, and a code sample illustrating its use. These design patterns can be used regardless of programming language and test tool. As such, specific test tool code is omitted in this paper and for consistency examples are given in Java, using the JUnit testing framework. This paper aims to illustrate that test code should be treated with the same importance as application code. This assertion has the following consequences:

Using application development design principles, automated test development can become a well factored art form that is adaptable to application changes. This adaptability results in lowered maintenance costs and easier test creation.

Posted by Misha Rybalov at 08:34 PM | Comments (0)

January 02, 2005

2. Introduction

Automated testing offers numerous benefits to any software organization, including finding defects cost-effectively early in the development cycle, providing rapid feedback, and giving developers the courage to refactor their code. With the recent popularization of Test Driven Development,1 automated testing has also become a design tool for application development.

Ironically, with all the attention paid to the benefits of automated testing, the actual art of designing and writing automated tests is often assumed to be taken care of by the test tool or the test writer. The principles of simple design, refactoring, and design patterns that are so common in software development are under utilized when it comes to test development.

While unit testing involves very simple method calls and checking for expected results, customer tests are significantly more complicated. It is quite common to have a single customer test scenario involving dozens of navigations and checks which span multiple screens and modules. The consequences of a haphazard approach to automated customer testing can result in massive code duplication, tests that are cumbersome to create, costly to maintain, and buggy.

This paper presents a catalog of design patterns that facilitate the creation and maintenance of automated customer tests. Each pattern consists of 1) a definition, 2) a discussion of the pattern details highlighting its suitability for customer test automation, and 3) a code sample illustrating its use.

This paper uses a fictional web-based shopping cart and catalog application to demonstrate how each pattern can be implemented. We will test the application's capabilities of searching for, adding, and removing items from the cart, as well as checking out and processing coupons.

Test Tools and Code

The design patterns introduced in this paper are not test tool specific in any way. One of the goals of applying these patterns is to decouple test code from the test tool code. For our purposes, we will use the JUnit testing framework2 to demonstrate the application of these patterns. These tests could just as easily have been written with another object-oriented testing framework (see Extreme Programming Software for a list of object-oriented testing frameworks3).


Customer Tests

This paper uses the term "customer tests" to refer to tests that exercise the application from an end-user's perspective. These types of tests are also referred to as functional or acceptance tests. Customer testing can be performed through a web browser (e.g. Internet Explorer, Netscape), an operating system's graphical user interface (e.g. Windows, Unix, MacOS), or any other user interface (e.g. Java, .NET). The patterns presented in this paper need not be limited to customer tests - they can be successfully applied to unit tests as well. This paper, however, focuses on customer tests because they are inherently more complex than unit tests and thus will derive the greatest benefit from design patterns.

Why Use Design Patterns to Automate Customer Tests?

Design patterns are used "to build flexible, reusable software."4 An automated test is, in essence, a program that checks another program. Consequently, it is vulnerable to the same design problems and entropy as application code. These vulnerabilities, often referred to as "code smells,"5 include test code duplication, tight coupling between the application and tests, and long test methods.

Traditionally, design patterns were not used for test code because test code was considered more simplistic and subordinate to application code. Also, most test tools generate customer test code through a record-and-playback mechanism (i.e. they record all user actions as the tester navigates through the application, create the corresponding code, and then play it back as the test), which encourages test developers to use the tool generated code without modifications. The challenge with both of these approaches is that tests become more difficult to maintain as the application changes over time. Design patterns help us minimize maintenance costs by making tests easier to update and more adaptable to application changes.

The patterns presented in this paper have been successfully used on real projects. These patterns can be applied in any object-oriented language such as Java, C++ or C#. They can also be used to refactor auto generated code from a commercial testing tool, but the mostly procedural nature of tool languages yields less benefit than an object-oriented language.

Posted by Misha Rybalov at 05:25 PM | Comments (0)