It is widely recognised that testing software can be an enormously challenging task, but why is this? The answer to this question is not straight forward; software is complex, code paths are long and the numbers are not on the tester’s side…

Consider a simple software requirement to determine if the result from the division of two natural numbers is itself a natural number. Using a specification language, such as the Z notation, this requirement may be expressed formally, as follows.

Now let us consider an eight-line Java code implementation from a software development engineer.


public boolean isDivWhole(int m, int n) {
  int res = m / n;
  if (res * n == m) {
    return true;
  } else {
    return false;
  }
}


From studying the specification, a test engineer will make key observations. First (being a tester), they will note a number of likely failure scenarios, for example, there is a specification constraint to operate on the set of natural numbers (positive whole numbers 1,2,3,…). Our simple specification is nondeterministic with respect to behaviour outside of this range of numbers. However, it is desirable that any implementation will provide appropriate error handling, promoting software resilience and removing the danger of fault aggregation which may lead to catastrophic failure.

The second observation is the most obvious; valid inputs must yield the correct answer, in other words ‘it must work’. To test this rigorously would require the checking of every possible input value, which in this case, is the whole set of natural numbers, and theoretically infinite. Even where constraints are imposed by hardware or programming languages (for the Java this will be the range 0 – 2147483647), this still leads to an impractical number of test cases. If it were possible to execute and analyse one million test permutations each second, this would still lead to a test cycle spanning more than 145,000 years. Most software testers will consider the execution of one million tests per second to be optimistic, while almost anyone will consider 145,000 years’ effort to be expensive for just eight lines of code.

More subtlety, the test engineer may seek to establish if the implementation is defensively coded, questioning how gracefully the module will handle conditions outside of the design parameters. In the above example, it is easy to answer such questions by analysing the code provided. This type of analysis is more commonly referred to as white-box-testing and requires a degree of programming skill, dependant on the code in question. Obviously, white-box-testing will become more intensive and therefore expensive as the volume of code increases. As a consequence, larger units of software are generally tested using the specification as the basis for test case design (commonly referred to as black-box-testing techniques). Usually, pragmatic use of all techniques will yield the greatest test impact.

Modern software products are measured in Millions of Lines Of Code (MLOC). In this context, the eight-line module above will be a small part of the overall solution and will likely interact with many other code modules. This adds another dimension to the problem and as a result, further multiplies the total number of valid test cases.

Whichever technique is employed, clearly some strategy is required to constrain the required test cycle into a realistic timeframe. This can only be achieved by accepting some degree of risk. A test engineer has two main approaches for this problem. First, following the principles of Myers¹, it is possible to reduce the number of required test cases by employing techniques such as Boundary-value Analysis and Equivalence Partitioning. Second, the test engineer can refine testing scope by the notion of customer relevance; the test engineer will seek to align the focus of testing efforts with expected patterns of customer utilisation.

In summary, it is very easy for testing to become exhausting. The deployment of software solutions are often time sensitive, and as a result test schedules will likely be constrained in some way. As testers we must spend resources wisely, identify and prioritize the right things to focus on. Risk can be a very useful tool in achieving these aims, but only if we harness it appropriately.

---

¹Glenford J Myers, The art of software testing, ISBN 0-471-04328-1

Tags: risk, techniques, z

This entry was posted on Tuesday, January 29th, 2008 at 19:18 by Richard Coppen. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.