Recently my car has been unhappy – and not meeting the “service level agreement” I expect from it. I realised I was a software tester when I began defining test cases to determine what was wrong with it… which at least highlighted several parallels (and lessons!) between the “traditional” and software engineering disciplines.

Error paths: expect the unexpected

I’m testing an error path in my car: something unexpected has happened due to a mechanical failure. A good analogy in software is that a back-end database has failed.

We need to test error paths – to ensure that our software products handle problems with minimal disruption to our customers. We need to understand what potential failures exist, to define acceptable outcomes (e.g., the main system should cope, albeit at reduced performance) and test them. We must also investigate the “serviceability” aspect – that is, how easy is it to diagnose and fix the initial problem? Finally, once we do fix the problem, we need to ensure the system recovers completely, rather than being left in a messed up state.

Test cases

I defined and executed two test cases to investigate the car issues. I will call them “load” and “stress” test cases – and we’ll return to this subject in future posts, as the distinction between load and stress is important.

• Load test
  Definition: Drive car up hill, with 30% throttle.
  Expected result: Car cruises up hill smoothly.
  Actual result: Car hesitates and does not accelerate.

• Stress test
  Definition: Drive car up hill, with 100% throttle.
  Expected result: Car accelerates smoothly and rapidly up hill. (I wish!)
  Actual result: Car hesitates even more. Engine management light flashes: “Seek mechanical assistance ASAP”.

Serviceability

These tests highlight a critical area – serviceability. The flashing warning light is effectively the car’s Engine Control Unit (ECU) saying, “I’ve detected a very nasty problem, and I’ve stored it in my log”. Great – I can’t read that log without spending money on a garage visit. Unnecessary costs . In software – we must ensure that failures have informative error messages – so we can do something about it without engaging consultants unnecessarily. Note this has a positive benefit on the software vendor too: the servicing burden is reduced.

Fixing

After some investigation, fingers crossed: I’ve found the problem. One of the HT leads was failing, causing a weak spark on one of the cylinders (leaving with only 75% design capacity). At least I had some redundancy in the engine, so if one cylinder fails the car still drives. Note the clear parallels with “high availability” in software – a vital component of many middleware applications.

Incidentally – back to serviceability. I suspect the warning light was pointing to a “downstream” symptom (probably an unexpected oxygen content in the exhaust) rather than the immediate source of the problem. If I had read the ECU log, I might have gone looking for the problem in the wrong (expensive) area…

Summary

Things go wrong in cars and in software – it’s unavoidable. What we can do as testers is ensure that we have the most painless resolution. Three areas to consider when testing software are:

1. Ensure failures do not escalate into critical problems (e.g., through redundancy).
2. Ensure diagnostics are useful – check error messages are sensible and point to the problem! Avoid downstream symptoms: detect problems as early as possible.
3. Ensure that on fixing a problem, the software recovers.

We’ll have more discussions on parallels between traditional and software engineering in future posts.

Tags: error, industry, load, parallel, stress

This entry was posted on Monday, January 28th, 2008 at 17:58 by Ben Bakowski. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.