Just been having a chat with a new grad who is working in one of our System Test teams.
We were talking about load, stress and performance testing and I was trying to articulate the reasons behind having a workload with peaks and troughs during the day (in the same way that a customer would) rather than our usual approach to load which is run at a constant high level for a long period. However, I was struggling with a justification as to why this might be a good idea and how it would find a different class of defects.

Then I remembered the ‘Top Gear’ episode when Hammond was trying to drive a Formula 1 car around a track. Apparently F1 cars are designed to go flat out – all the time – and are pretty good at this. What Hammond discovered to his horror was that when you don’t drive them flat out then you can get into all kinds of trouble.  Cornering was a particular problem. Flat out the tyres are warm and stick to the track, the brakes responsive and the down force of the car helps it round. Anything less than flat out and as Hammond found you are all over the place.

So in this case if your system is the car and you only use Lewis Hamilton to test it then there are a lot of defects that will get missed. Fortunately few people drive F1 cars slowly on the road, but this would be a severe problem if the system under test was something like a mini.

So moral of the day, remember for every Lewis Hamilton tester you have you need at least one Hammond.

I started writing a comment to Ben Bakowski’s last post on “Stressful testing…“, but decided to turn it into a full post. I was thinking that a key difference between testing a “real world” object such as a car, and something like software, is that the limits aren’t always as clear. A car might have five seats, and might use roads where the speed limit is 70mph – these are concrete numbers, which allow you to make the distinction more easily. If I have a piece of software (especially a piece of middleware), then it’s tricky to know the limits. How many records can my database handle? How big can they be? How many users at once?

When testing middleware, I think you’re usually forced to ask yourself: what do I expect someone to do with this? It’s largely about guesswork and experience with existing customers. I think that an interesting idea to explore, is whether it is appropriate or advisable to set artificial limits on a product’s maximum load. Setting artificial limits could provide actual boundaries to the testing that needs to be done. I can think of a few “real world” analogies:

• Putting a “maximum load” sticker on a lorry – ignore at your own risk.
• Putting a fixed number of seats in the car – “we could fit eight seats in here, but we’ve only put in five, so only try and put five people in it”.
• Fitting a speed limiter (i.e. a hard limit).

We could do the same in software: we could write out log messages when a particular level of load is exceeded; we could put in hard limits to the number of records in a database and the number of concurrent users. The key thing when doing this, I think, would be provide reasonable limits which won’t impede your customers, but at least “frame” your testing. For those die-hard customers who absolutely can’t do without that ten millionth database record, you could always offer a special option to remove the limiters, subject to additional testing, or acceptance of the additional risk.

What do you think?  Would this cause as many problems as it fixes?  Or would it enable you to focus on the testing that matters?

In an earlier blog I mentioned that there is a distinction between “load” and “stress” testing which is worth further clarification. This is important – as knowing which regime you’re in helps you understand a test’s value.

Crudely, load and stress testing involve pushing software/hardware at different levels, the latter “harder” than the former. Jon Tilt suggested a good analogy based around testing a passenger car. In a good demonstration of “reuse”, I’ve shamelessly stolen the analogy here…

(more…)

Recently my car has been unhappy – and not meeting the “service level agreement” I expect from it. I realised I was a software tester when I began defining test cases to determine what was wrong with it… which at least highlighted several parallels (and lessons!) between the “traditional” and software engineering disciplines.

(more…)