I started writing a comment to Ben Bakowski’s last post on “Stressful testing…“, but decided to turn it into a full post. I was thinking that a key difference between testing a “real world” object such as a car, and something like software, is that the limits aren’t always as clear. A car might have five seats, and might use roads where the speed limit is 70mph – these are concrete numbers, which allow you to make the distinction more easily. If I have a piece of software (especially a piece of middleware), then it’s tricky to know the limits. How many records can my database handle? How big can they be? How many users at once?

When testing middleware, I think you’re usually forced to ask yourself: what do I expect someone to do with this? It’s largely about guesswork and experience with existing customers. I think that an interesting idea to explore, is whether it is appropriate or advisable to set artificial limits on a product’s maximum load. Setting artificial limits could provide actual boundaries to the testing that needs to be done. I can think of a few “real world” analogies:

• Putting a “maximum load” sticker on a lorry – ignore at your own risk.
• Putting a fixed number of seats in the car – “we could fit eight seats in here, but we’ve only put in five, so only try and put five people in it”.
• Fitting a speed limiter (i.e. a hard limit).

We could do the same in software: we could write out log messages when a particular level of load is exceeded; we could put in hard limits to the number of records in a database and the number of concurrent users. The key thing when doing this, I think, would be provide reasonable limits which won’t impede your customers, but at least “frame” your testing. For those die-hard customers who absolutely can’t do without that ten millionth database record, you could always offer a special option to remove the limiters, subject to additional testing, or acceptance of the additional risk.

What do you think?  Would this cause as many problems as it fixes?  Or would it enable you to focus on the testing that matters?

Why do we test software? This may seem an obvious question: everyone has at some point experienced the frustration of error-prone, or “buggy”, software. As software becomes increasingly pervasive in modern life, simple frustrations can quickly become much more serious. Just as traditional engineering disciplines involve testing of generated artifacts to ensure fitness for purpose, safety and durability, the same requirements are true for software, even though software engineering may seem abstract to a traditional engineer.

We are a group of senior software testers working in the IBM Hursley software labs in the UK. We work on large middleware projects, across multiple testing disciplines. This is an unofficial collection of our thoughts and ideas on software testing, based around our philosophy of risk-based testing. We want to share our ideas, and expand them based on other people’s thoughts. We feel that a blog is an ideal format for this: please contribute.

Arthur, Ben, Jon, Richard and Scott