If you are familiar with the BBC’s Little Britain comedy series, you will have probably seen David Walliams performing a Mr Mann sketch. Mr Mann is a very demanding customer, he has ludicrously specific requirements and no ability to compromise. Much to the annoyance of shop keeper Roy, Mr Mann is prepared to simply ‘wait’ – presumably forever – until he gets what he wants.

As Software Engineers, the very nature of our job means that we work with very specific sets of requirements. As software testers we have a tendency to blend these together into complex test scenarios with the sorts of prerequisites and demands that would make even the likes of Mr Mann seem easy to please. Sadly, when we don’t get all the pieces we need in one go, the temptation to wait for the rest of them to arrive can become overwhelming.

It is important not to forget that each and every test scenario is designed for good reason and will no doubt reveal something fundamental about the item under test. However, we must keep this end goal in context. If we can’t proceed with Scenario S as is, we must challenge ourselves and re-examine our test tactics to keep things moving. This is vital if we are to keep lowering the risk to the parent project.

OK, so this approach won’t always pay off, it’s a bit like trying to avoid a traffic jam – sometimes you just can’t. It is also important that we don’t become engrossed with spinning-plates or distracted with investigating the trivial low-value areas.

Sometime ago I had a track driving lesson, the instructor told me “Whatever you’re doing, it isn’t working. Try something different – anything will do!”. He certainly had a way of emphasizing this message, and I didn’t much appreciate the feedback at the time, but maybe he had a valuable point?

It is widely recognised that testing software can be an enormously challenging task, but why is this? The answer to this question is not straight forward; software is complex, code paths are long and the numbers are not on the tester’s side…

(more…)

If a team of people produce something, whose fault is it if the quality is poor? After all, a tester’s job is, ostensibly, to find defects in a product. A poor quality product must be the result of the testers not doing well enough, right? After all, test teams are often referred to as “Quality Assurance” teams. Well, I’d say that the testers aren’t the ones making the quality low, and everyone involved in the project is responsible for the quality, including designers, developers, testers and managers. By discovering a defect, a tester is merely saying that a problem exists, and is highlighting a risk. I think the question we should be asking, is who is responsible for the risk?

(more…)

Recently my car has been unhappy – and not meeting the “service level agreement” I expect from it. I realised I was a software tester when I began defining test cases to determine what was wrong with it… which at least highlighted several parallels (and lessons!) between the “traditional” and software engineering disciplines.

(more…)

In discussion with a friend it hit me that one of the biggest obstacles in going Agile is trying to convince people to deliver “little and often”.

What this means is that you deliver just enough code to get the feature working and leave out any “I’m gonna put this here for later use” code. You also accept that this may lead to re-writes of that code in later iterations. This implies that your code is not designed to be extensible from the outset.

On the opposite side is that of having a long delivery time for your feature. This gives you the flexibility of spending a considerable time up front designing the infrastructure to support any code that will follow in that release. Unfortunately with this approach you get the inherent problems of having a long waterfall release; little customer interaction and often it’s too late.

Now, what does this mean for Test? Well, going back to the first statement in this post. I think in the transition to Agile it can be really tough to break a developers’ mentality to write extensible code. Thus from a Test perspective you get elements of both extensible and time-to-market, meaning…

Too much to test in too little time.

Depending on your point of view, risk is either a good or bad thing. The pessimist sees the doom and gloom of future problems, the optimist delights in the opportunity, the naive ignore risks completely. As a Test Architect, I see risk as one of my most powerful tools in driving a successful test cycle. I have a ‘virtual risk lever’ that I can pull or release depending on the situation. I can use risk to justify priorities and resource, focus attention and articulate a clear business view of our testing. So what do I mean by ‘risk’ in a software testing context?

Taking a simple definition: Risk is the possibility of an event occurring that will have an impact on the achievement of objectives. Risk is measured in terms of impact and likelihood.

To give an example, if a tightrope walker is walking along a wire one foot above the ground then, regardless of the probability, the impact of falling off is minimal. A sprained foot at worst. This could be deemed a low risk activity. If the same performer walks along a wire one thousand feet from the ground the probability of falling off essentially remains the same, but the impact of falling and receiving major injuries is far greater. In this case, broken bones and worse are a certainty. This would then be considered a high risk activity. So we can see that for the same simple activity the risk can vary depending on the situation.

So how can we use risk to drive our testing?

(more…)

Why do we test software? This may seem an obvious question: everyone has at some point experienced the frustration of error-prone, or “buggy”, software. As software becomes increasingly pervasive in modern life, simple frustrations can quickly become much more serious. Just as traditional engineering disciplines involve testing of generated artifacts to ensure fitness for purpose, safety and durability, the same requirements are true for software, even though software engineering may seem abstract to a traditional engineer.

We are a group of senior software testers working in the IBM Hursley software labs in the UK. We work on large middleware projects, across multiple testing disciplines. This is an unofficial collection of our thoughts and ideas on software testing, based around our philosophy of risk-based testing. We want to share our ideas, and expand them based on other people’s thoughts. We feel that a blog is an ideal format for this: please contribute.

Arthur, Ben, Jon, Richard and Scott