« 13. Architectural Patterns | Main | 11. Design Principle #3 - Don't Repeat Yourself (DRY) »
November 15, 2004
12. Design Principle #4 - Multiple Failures
Create a mechanism to allow a customer test to continue executing after a non-critical failure.
Traditionally, unit testing tools such as JUnit fail a test automatically after encountering the first failure. This does not impede unit testing, since most unit tests execute only a few steps where each is dependent on the previous one. Customer tests, however, often execute dozens of steps and can have steps that are independent of each other.
When testing a module that is buried deep within an application, there are many steps involved to reach the module of interest. If one of those initial steps fails, all subsequent steps would not be executed. This includes all test steps related to the module of interest. Bugs beyond this first failure can be masked within the application. This also introduces inefficiency since each bug must be found and fixed before the next one can be found. Finding multiple bugs at a time increases efficiency and allows bugs to be addressed in a priority sequence.
If the initial failure was due to incorrect information but that information is not required by the module of interest, then the test could potentially continue validly executing and performing additional steps. Thus, when it comes to customer tests, an approach is needed to allow multiple failures.
The Multiple Failures pattern, however, is more difficult to implement than the other patterns, since JUnit does not support multiple failures. In collaboration, the author has modified the JUnit testing framework to add this functionality.
In this example, there is a problem with the specials page such that incorrect items are displayed. This page is mandatory in the navigation path of searching for and adding an item to your cart. Therefore, without a multiple failures mechanism, the adding function would not be tested because the test would fail and stop executing on the specials page. However, the specials page is not integral to adding an item to the cart. By allowing multiple failures, the specials page failure can be logged, but the test can proceed and test the function of adding an item to the cart. Figure 13 illustrates how the multiple failure mechanism is applied.
Figure 13: Setting up multiple failures mechanism.
import junit.framework.*;
public abstract class ShoppingCartTemplate extends TestCase {
public ShoppingCartTemplate(String pName) { super(pName); }protected void setUp() {
login();
gotoSpecialsPage(); //navigate to the specials page
setStopOnFail(false); //JUnit extension - test will continue if check fails
checkSpecialsPage(); //if this check fails, the test will not stop
setStopOnFail(true); //JUnit extension - test failure will now stop the test
searchForItem1();
cartAction1(); // <------ test hook 1
searchForItem2();
cartAction2(); //<-------- test hook 2
logout();
}
abstract protected void cartAction1();
abstract protected void cartAction2();
}
The setStopOnFail() method demarcates when to start or stop the multiple failure mechanism. Notice that we demarcated only the checking of the specials pages and not the navigation. This is because a problem with the application's navigation cannot be overcome by our test. On the other hand, a checking failure does not prevent us from navigating to other parts of the application and performing our add to cart test. By demarcating which parts of a test allow multiple failures, we can exercise more parts of the application.
Posted by Misha Rybalov at November 15, 2004 01:34 PM