Stressful testing…
In an earlier blog I mentioned that there is a distinction between “load” and “stress” testing which is worth further clarification. This is important - as knowing which regime you’re in helps you understand a test’s value.
Crudely, load and stress testing involve pushing software/hardware at different levels, the latter “harder” than the former. Jon Tilt suggested a good analogy based around testing a passenger car. In a good demonstration of “reuse”, I’ve shamelessly stolen the analogy here…
Suppose we wanted to test a standard passenger car. Let’s then suppose a “normal” journey is specified as a drive from London to Edinburgh at 65-70 mph for four occupants.
Load: demonstrate the car can be used in everyday work, with no long term problems.
Five sequential journeys, 1 driver, 3 passengers, 70 mph.
Stress: demonstrate the car can, at peak times, be worked harder to meet short-term demands.
One journey, 1 driver, 3 passengers, 85 mph.
One journey, 1 driver, 5 passengers, 70 mph.
Why do we have a distinction? Well, load testing is representative of typical usage, and hence vital to help demonstrate the car is fit for purpose. However, for stress testing - what *is* an acceptable outcome? It would be good to see the car handle these higher workloads - but at some point, the car will probably overheat. Is this a failure? The true answer is: perhaps not. Overheating could well be acceptable if you’re pushing the car beyond its design specifications - after all, it’s definitely preferable to catastrophic engine failure. If the engine’s overheated, all the driver needs to do is wait a bit and he’s back on his way.
It’s the same with software: load and stress testing should be viewed differently. If you think you’re in the load regime, when actually you’re in stress, you may waste hours trying to investigate “strange” software failures when in fact you’re really observing acceptable behaviour.
A good tip to follow: If you find yourself “willing” a load test to pass - take a step back and try and understand whether you have crept into the stress regime, and hence whether you should reset your expectations.
March 4th, 2008 at 23:08
Good reuse Ben
Stress is probably the most misunderstood area of system test. Most people talk about stress when they mean load. For me stress is going beyond the acceptable limits of the product, but with the assumption that failure will be graceful (if not deterministic).
I also think we over-stress our products. To extend your (our?) example a typical stress test would be to drive to John O’Groats, at 120mph with 7 people on board. Of course the car is going to break, but is it really going to help improve the product? Likewise, attacking the same car with a sledgehammer is going to break it, but is that useful information? If we are going to do stress testing we must ensure that all the tests are seeking to add value to the product.