« October 2004 | Main | December 2004 »
November 29, 2004
10. Design Principle #2 - Independent Test
Each test leaves the application in the same pre-defined state, irrespective of whether the test passed or failed.
In order to ensure test integrity, it is important that tests be unaffected by the outcome of other tests. Each test expects to start from a constant application state. This requires each test to de-initialize to this constant state upon completion or failure. If this does not occur then subsequent tests will fail to execute. Unit testing frameworks such as JUnit recognize the importance of this principle and have mechanisms to initialize and de-initialize each test in order to keep the application in a consistent state (i.e. using setUp() and tearDown() methods).
For our sample application, it is important that each test start on the login page, with an empty shopping cart. This necessitates a mechanism for emptying the shopping cart and logging out at the end of each test or upon test failure. If we use the Template pattern, we can insert these common steps of initialization and de-initialization, right into the Template, thereby relieving test writers from having to remember to include these steps in every test and ensuring tests stay independent.
Figure 10: Common initialization and de-initialization steps for each test case.
protected void setUp() {
login(); // <------ common setup code for all test cases
searchForItem1();
cartAction1();
searchForItem2();
cartAction2();
emptyCart(); // <------- common tear down code for all test cases
logout();
}
Posted by Misha Rybalov at 08:21 PM | Comments (0)
November 22, 2004
11. Design Principle #3 - Don't Repeat Yourself (DRY)
Minimize test code duplication to lower test maintenance costs.
The evils of code duplication have been documented extensively in software development literature.10 The same principles that apply to application development also apply to test development. Namely, just as it becomes a maintenance problem to modify application code in multiple places, the same problem occurs when tests need modification in multiple places. Many of the patterns mentioned in this paper (e.g. Template, Object Genie, DTO, Transporter) minimize code duplication by consolidating common test code into one place instead of duplicating it. The advantage of this approach is that it becomes easier to update tests as the application changes, which in turn lowers the maintenance costs of tests.
To test the coupon feature of the application, we need tests for submitting a valid coupon, an expired coupon, and an invalid coupon. In order to test these scenarios, we need to load our cart with items to which the coupon would apply. This would be the setup part of the test (see Independent Test pattern), since before we could test the checkout process, we'd have to have a cart with items in it already. If we don't adhere to DRY, we would simply copy and paste previous test code that searched for an item(s) and inserted it into the cart. Figures 11 and 12 illustrate non-DRY and DRY approaches, respectively.
Figure 11: Repeating test code for coupon test cases.
import junit.framework.*;
public class CouponTest extends TestCase {
public CouponTest(String pName) { super(pName); }
public static Test suite() { return new TestSuite(CouponTest.class); }
public void testValidCoupon() {
login();
searchForItem1();
addItem1ToCart();
searchForItem2();
addItem2ToCart();
gotoCheckout();
applyValidCoupon();
checkCouponAccepted();
}
public void testExpiredCoupon() {
login();
searchForItem1();
addItem1ToCart();
searchForItem2();
addItem2ToCart();
gotoCheckout();
applyExpiredCoupon();
checkCouponRejected();
}
public void testInvalidCoupon() {
login();
searchForItem1();
addItem1ToCart();
searchForItem2();
addItem2ToCart();
gotoCheckout();
applyInvalidCoupon();
checkCouponRejected();
}
//tool-specific implementations of methods (e.g. login(), addItem2ToCart(), etc.)
}
This non-DRY approach is recommended by some due to its inherent simplicity and explicitness.11 The argument is that if a test developer tries to get too fancy with their test code, they will have to start writing tests for the test code, which can quickly get out of hand. However, writing more maintainable code does not mean that the code has to get more complex.
Using DRY, we have several options: 1) use the Template pattern to extract the act of filling up your cart to a super class, 2) use an Object Genie to grant us a cart with items already in it, 3) extract the setup code into a set up method so that each test case uses the same setup code, and 4) combine the above methods.
Template Pattern
Using a Template pattern, we can extract all the setup code into a super class and have each subclass implement a few small methods. All the common code is encapsulated in the super class and the subclasses don't have to concern themselves of dealing with it. The disadvantage, though, is that a separate subclass has to be created for each test case.
Object Genie
An Object Genie can grant us a cart with items in it anytime we want it. The advantage of this technique is that we are not forced to use subclasses. We just request the cart with items in it at the appropriate time.
Setup Method
JUnit supports Setup Methods that allow a test writer to place all common setup code into one place. The setup code gets executed before each test case is run. This is a good start to avoiding duplication, but often the code that gets factored out into the Setup Method could be duplicated in other Setup Methods for other tests. In our example, the setup code of filling a cart with items is needed by other tests, as well. For instance, to test cart manipulation functions (adding, removing, modifying items in the cart) we would need to have a cart with items in it as setup code. This implies that the setup code should be located in a more central location, as outlined in the previous two examples.
Combining Patterns
Filling up a cart is a common test step that is useful to many customer tests. The three pattern options to address this step can be used in combination to yield a greater benefit. Figure 12 combines the use of an Object Genie and a Setup Method. Other combinations are possible as well, including using all three patterns. We use the Setup Method to login, call the Object Genie, and go to the checkout page. The Object Genie returns a shopping cart with items in it. The result is that each test case only has to worry about entering a specific coupon and checking whether the coupon was accepted or rejected.
Figure 12: Consolidating test code using Object Genie and Setup Method.
import junit.framework.*;
public class CouponTest extends TestCase {
public CouponTest(String pName) { super(pName); }
public static Test suite() { return new TestSuite(CouponTest.class); }
protected void setUp() {
login();
ShoppingCart shoppingCart = CartGenie.getNonEmptyCart();
gotoCheckout();
}
public void testCheckoutWithValidCoupon() {
applyValidCoupon();
checkCouponAccepted();
}
public void testCheckoutWithExpiredCoupon() {
applyExpiredCoupon();
checkCouponRejected();
}
public void testCheckoutWithInvalidCoupon() {
applyInvalidCoupon();
checkCouponRejected();
}
//implementations of methods (e.g. login(), addItem2ToCart(), etc.)
}
We have now significantly reduced the size of each test, eliminated test code duplication and reduced future test code maintenance costs.
Posted by Misha Rybalov at 05:25 PM | Comments (0)
November 15, 2004
12. Design Principle #4 - Multiple Failures
Create a mechanism to allow a customer test to continue executing after a non-critical failure.
Traditionally, unit testing tools such as JUnit fail a test automatically after encountering the first failure. This does not impede unit testing, since most unit tests execute only a few steps where each is dependent on the previous one. Customer tests, however, often execute dozens of steps and can have steps that are independent of each other.
When testing a module that is buried deep within an application, there are many steps involved to reach the module of interest. If one of those initial steps fails, all subsequent steps would not be executed. This includes all test steps related to the module of interest. Bugs beyond this first failure can be masked within the application. This also introduces inefficiency since each bug must be found and fixed before the next one can be found. Finding multiple bugs at a time increases efficiency and allows bugs to be addressed in a priority sequence.
If the initial failure was due to incorrect information but that information is not required by the module of interest, then the test could potentially continue validly executing and performing additional steps. Thus, when it comes to customer tests, an approach is needed to allow multiple failures.
The Multiple Failures pattern, however, is more difficult to implement than the other patterns, since JUnit does not support multiple failures. In collaboration, the author has modified the JUnit testing framework to add this functionality.
In this example, there is a problem with the specials page such that incorrect items are displayed. This page is mandatory in the navigation path of searching for and adding an item to your cart. Therefore, without a multiple failures mechanism, the adding function would not be tested because the test would fail and stop executing on the specials page. However, the specials page is not integral to adding an item to the cart. By allowing multiple failures, the specials page failure can be logged, but the test can proceed and test the function of adding an item to the cart. Figure 13 illustrates how the multiple failure mechanism is applied.
Figure 13: Setting up multiple failures mechanism.
import junit.framework.*;
public abstract class ShoppingCartTemplate extends TestCase {
public ShoppingCartTemplate(String pName) { super(pName); }protected void setUp() {
login();
gotoSpecialsPage(); //navigate to the specials page
setStopOnFail(false); //JUnit extension - test will continue if check fails
checkSpecialsPage(); //if this check fails, the test will not stop
setStopOnFail(true); //JUnit extension - test failure will now stop the test
searchForItem1();
cartAction1(); // <------ test hook 1
searchForItem2();
cartAction2(); //<-------- test hook 2
logout();
}
abstract protected void cartAction1();
abstract protected void cartAction2();
}
The setStopOnFail() method demarcates when to start or stop the multiple failure mechanism. Notice that we demarcated only the checking of the specials pages and not the navigation. This is because a problem with the application's navigation cannot be overcome by our test. On the other hand, a checking failure does not prevent us from navigating to other parts of the application and performing our add to cart test. By demarcating which parts of a test allow multiple failures, we can exercise more parts of the application.
Posted by Misha Rybalov at 01:34 PM | Comments (0)
November 08, 2004
13. Architectural Patterns
Architectural patterns dictate the organization of test code into a framework. This group consists of the Three-Tier Testing Architecture.
Posted by Misha Rybalov at 12:40 PM | Comments (0)
November 07, 2004
14. Architectural Pattern - Three-Tier Testing Architecture
Create three layers of test code: Tool Layer, Application Testing Interface Layer and Test Case Layer.
It is a well established idea in distributed object architecture to divide an application into separate tiers. The first tier encapsulates the presentation logic, the second tier the business logic, and the third tier the data storage logic.12 Using this paradigm, application maintenance costs are reduced since components inside each tier can change without impacting other tiers. The same principle can be applied to customer testing architecture.
Test code can be separated into three layers: the Tool Layer, the Application Testing Interface (ATI) Layer and the Test Case Layer. Each layer has a specific responsibility, with the overall goal of reducing the maintenance costs of tests and facilitating new test creation.
Tool Layer
The Tool Layer's responsibility is to provide an extensive Application Programming Interface (API) to facilitate test writing. For instance, the tool can allow tests to find and interact with objects such as buttons, forms, and tables. Each tool is generic such that it can be used to write tests for various applications of a specific type (e.g. web, .NET or Java). Some tools have capabilities to test applications of several types.
Test tools usually fall into two types: record-playback (e.g. QuickTest13 and programmer-oriented. Using a record-playback tool, test developers can quickly start recording tests and playing them back. However, the code that is generated by the tool is usually not object-oriented and uses a proprietary language. Tools that are programmer-oriented are meant to be used by someone with programming experience. There is usually no record and playback mechanism because test developers are expected to write the tests themselves. The advantage of these tools is that they use a standard object-oriented language, which makes them very suitable to using design patterns.
Application Testing Interface (ATI) Layer
The ATI Layer14 is responsible for consolidating all functions common to multiple tests. Every application will have a specific ATI Layer. For instance, each application is sufficiently unique as to require different navigations and assertions. Thus, the common ones contained in this layer would vary between applications. The ATI Layer is in effect a utility service provider for individual tests. It offloads the majority of the work that each test would otherwise be required to do.
The Template, Domain Test Object, Object Genie, and Transporter patterns are all examples of ATI layer code. The ATI Layer uses the tool (from the Tool Layer) to implement each pattern's details. If the tool or the application changes, then the corresponding ATI code would need to be updated to accommodate the changes. The ATI layer is the primary factor in lowering maintenance costs, since a well designed ATI will encapsulate many common navigations and checks that are used by test cases in the Test Case Layer.
Test Layer
The purpose of the Test Layer is to outline specific inputs and expected outputs for each test. The Test Layer relies heavily on the ATI Layer and as a result, is quite brief. Ideally, the Test Layer contains only data unique to each test. For this reason, the tests are insulated from changes in the application. Examples of Test Layer code are the subclasses of a Template, and the tests that use an Object Genie, Transporter, or DTO.
Posted by Misha Rybalov at 08:12 PM | Comments (0)
November 01, 2004
15. Conclusion
Applications are constantly evolving, which poses serious maintenance problems for automated customer tests. Maintenance problems can be addressed by using design patterns and thus, treating test code with the same importance as application code. This paper presented a catalog of design patterns that demonstrate how to facilitate: test code reuse without duplication, test adaptability to application changes, creation of new tests, and maintenance of existing tests. To this end, the following nine design patterns were presented:
Group 1: Basic Patterns
- Template - consolidate sequence of steps to perform a test
- Object Genie - consolidate object creation and initialization logic
- Domain Test Object - consolidate knowledge of application's UI components
- Transporter - consolidate navigation logic
Group 2: General Design Principles
- Dumb Test - minimize tests' knowledge of the application
- Independent Test - maximize test independence from other tests
- Don't Repeat Yourself - minimize duplication of code between tests
- Multiple Failures - maximize the information obtained from each test
Group 3: Architectural Patterns
- Three-Tier Testing Architecture - separate test code into layers
The design patterns were categorized into three groups: Basic Patterns, General Design Principles, and Architectural Patterns. The basic patterns form the building blocks that the general design principles use to establish best practices. The architectural pattern dictates the organization of all test code into a reusable framework.
This paper has demonstrated how design patterns can be applied to increase the effectiveness of automated customer testing. This results in tests that consistently provide valuable and reliable feedback about an application. It is my hope that this information will be of benefit to others in their automated testing endeavors.
Posted by Misha Rybalov at 02:32 PM | Comments (0)