I started writing a comment to Ben Bakowski’s last post on “Stressful testing…“, but decided to turn it into a full post. I was thinking that a key difference between testing a “real world” object such as a car, and something like software, is that the limits aren’t always as clear. A car might have five seats, and might use roads where the speed limit is 70mph – these are concrete numbers, which allow you to make the distinction more easily. If I have a piece of software (especially a piece of middleware), then it’s tricky to know the limits. How many records can my database handle? How big can they be? How many users at once?

When testing middleware, I think you’re usually forced to ask yourself: what do I expect someone to do with this? It’s largely about guesswork and experience with existing customers. I think that an interesting idea to explore, is whether it is appropriate or advisable to set artificial limits on a product’s maximum load. Setting artificial limits could provide actual boundaries to the testing that needs to be done. I can think of a few “real world” analogies:

• Putting a “maximum load” sticker on a lorry – ignore at your own risk.
• Putting a fixed number of seats in the car – “we could fit eight seats in here, but we’ve only put in five, so only try and put five people in it”.
• Fitting a speed limiter (i.e. a hard limit).

We could do the same in software: we could write out log messages when a particular level of load is exceeded; we could put in hard limits to the number of records in a database and the number of concurrent users. The key thing when doing this, I think, would be provide reasonable limits which won’t impede your customers, but at least “frame” your testing. For those die-hard customers who absolutely can’t do without that ten millionth database record, you could always offer a special option to remove the limiters, subject to additional testing, or acceptance of the additional risk.

What do you think?  Would this cause as many problems as it fixes?  Or would it enable you to focus on the testing that matters?

In an earlier blog I mentioned that there is a distinction between “load” and “stress” testing which is worth further clarification. This is important – as knowing which regime you’re in helps you understand a test’s value.

Crudely, load and stress testing involve pushing software/hardware at different levels, the latter “harder” than the former. Jon Tilt suggested a good analogy based around testing a passenger car. In a good demonstration of “reuse”, I’ve shamelessly stolen the analogy here…

(more…)

Recently my car has been unhappy – and not meeting the “service level agreement” I expect from it. I realised I was a software tester when I began defining test cases to determine what was wrong with it… which at least highlighted several parallels (and lessons!) between the “traditional” and software engineering disciplines.

(more…)