Testing without limits
I started writing a comment to Ben Bakowski’s last post on “Stressful testing…“, but decided to turn it into a full post. I was thinking that a key difference between testing a “real world” object such as a car, and something like software, is that the limits aren’t always as clear. A car might have five seats, and might use roads where the speed limit is 70mph - these are concrete numbers, which allow you to make the distinction more easily. If I have a piece of software (especially a piece of middleware), then it’s tricky to know the limits. How many records can my database handle? How big can they be? How many users at once?
When testing middleware, I think you’re usually forced to ask yourself: what do I expect someone to do with this? It’s largely about guesswork and experience with existing customers. I think that an interesting idea to explore, is whether it is appropriate or advisable to set artificial limits on a product’s maximum load. Setting artificial limits could provide actual boundaries to the testing that needs to be done. I can think of a few “real world” analogies:
- Putting a “maximum load” sticker on a lorry - ignore at your own risk.
- Putting a fixed number of seats in the car - “we could fit eight seats in here, but we’ve only put in five, so only try and put five people in it”.
- Fitting a speed limiter (i.e. a hard limit).
We could do the same in software: we could write out log messages when a particular level of load is exceeded; we could put in hard limits to the number of records in a database and the number of concurrent users. The key thing when doing this, I think, would be provide reasonable limits which won’t impede your customers, but at least “frame” your testing. For those die-hard customers who absolutely can’t do without that ten millionth database record, you could always offer a special option to remove the limiters, subject to additional testing, or acceptance of the additional risk.
What do you think? Would this cause as many problems as it fixes? Or would it enable you to focus on the testing that matters?
Tags: limits, load, middleware, risk, stress
March 12th, 2008 at 15:10
Interesting insights Arthur.
I think, in order to have finite testing, rather than infinite testing, we should have limits. Right?
Sometimes I say “how do we test this new piece of stuff fully?”, often this is because I have no limits on how far this stuff is allowed to go. Writing this got me thinking, when I say limits, do I actually mean boundaries? Limits “on how far it can go”, boundaries “where can it go?” To take a word processor, limits are on how big a document can it handle, boundaries on what 3rd party embedded entries can a document handle.
I wonder also whether, if we have clearly defined boundaries and limits, we can auto-generate tests? Thereby removing the need to hand-write tests? The boundaries and limits could also be turned into documentation for the customer.
This is my thinking as a developer.
March 12th, 2008 at 15:25
I agree, and it does seems like it would be easier to do model-based testing if you could actually generate a finite model of the software under test.
March 24th, 2008 at 19:03
Some interesting thoughts!
One of the “dangers” when testing software is that when defining load testing, some sort of benchmark has to be defined. This can almost be too arbitrary - particularly if the software’s “new” with no real-life precedence of workloads.
If the benchmark is too low, real customer usage isn’t modelled correctly, but if it’s too high, then testers spend a disproportionate amount of time tweaking software and hardware trying to reach unattainable performance goals. Again, it’s a fine balance to achieve…