Testing v Checking
Had a great meeting this morning with a number of senior testers from Hursley.
The main theme of the discussion was around an article found by Russell Finn, ‘Testing v Checking‘ – where basically checking is doing what you are told and testing is doing what you feel should be done (verification v validation might be a better way of putting it).
We have evidence from one of our major products that 50% of our field reported problems are things no one had considered (not designed, coded or tested for). So whilst we are pretty good at the verification side of things, there’s a lot we miss becuase we are not testing to real stakeholder needs.
A solution we discussed was around exploratory testing where the emphasis is on learning about a system and improving it through testing.
We also discussed quality metrics and how they scale. Traditional methods of counting defects per 1000 lines of code are fundamentally flawed because they don’t consider the impact on the end user. One defect in a 100 line application might sound good, but what if that code is distributed across a million devices and each one needs an update?
Perhaps a better method would be ‘mean time to defect’ – some sort of measure around the time it takes an average end user to encounter a new defect.
Final thoughts were around the combative nature of a tester raising defect. Each time we raise a defect we are effectively telling the coder their baby is ugly. Is there a better way of doing this to get testers and coders working cooperating more? One suggestion was to change the term defect to ‘opportunities for improvement’ – not ideal I know but I like the sentiment. Comment if you have any better ideas.
January 16th, 2010 at 05:55
Hi there, _________ (your name here)…
Final thoughts were around the combative nature of a tester raising defect. Each time we raise a defect we are effectively telling the coder their baby is ugly. Is there a better way of doing this to get testers and coders working cooperating more?
Yes, there is. It’s important for you to get out of the defect business, and into the “possible problem” business.
Over the years, the notion of the tester’s role got corrupted from being in partnership with the programmers to being antagonists. That coincided with some ill-advised person deciding to give testing a grander title, and calling it “quality assurance”. Testers ceased to be an information service, and became the “gatekeepers of quality”.
We can get that back by stopping the pretense that we are the arbiters of what is and is not a bug. As testers, we don’t know the deep truth about what we see. Our oracles—the principles or mechanisms by which we recognize a problem—are heuristic, meaning that they’re more-or-less reliable, but ultimately fallible. When we think we see a problem, there could be some countervailing benefit that makes it less of a problem than we think; when we think we don’t see a problem, our oracles could be fooling us.
So: instead of saying that this or that is a defect, say that you’ve observed something that might be a problem. Explain why you think it could be a problem—say, that it’s inconsistent with similar products, or that it contradicted some reasonable expectation that a user might have, or that it’s inconsistent with some claim that some important person has made about the product via a requirements document. Since the inconsistency with similar products might be intentional, or the expectation might be reasonable from another perspective, or since the requirements document may be out of date, you don’t know for sure that you’re seeing a problem. Instead, say that you’re seeing a possible problem, and is that, Dear Programmer, okay with you?
Note that it’s not really the programmers’ job to decide whether something is a problem either. In fact, it’s the product owner’s job. So if you and the programmer disagree, the appropriate and immediate response is, “You may be right. I could be wrong. Let’s not argue about it; let’s let the product owner decide.” Report clearly and accurately why you think something might be a problem, and if the programmers disagree, acknowledge that disagreement (without necessarily agreeing). Learning how to say, “You may be right” is important. Learning how to believe it is even more important.
Perhaps a better method would be ‘mean time to defect’ – some sort of measure around the time it takes an average end user to encounter a new defect.
I’m not convinced that a crash, with data loss, after five hours of work is significantly better than a crash, with data loss, after two hours of work. Nor will your customers use the product in the same way. But remember: you don’t have to decide anything about quality metrics. Outline the problem, the threat that it poses to the value of the product, some compelling story about how one or more users might suffer harm or inconvenience or annoyance, and then let the product owner do his job: managing the project.
Feel free to let me know if there’s some other way in which I can be of service.
—Michael B.
January 18th, 2010 at 11:17
@Michael, I entirely agree that it’s not the tester’s role to be an arbiter of quality (also see “Who is responsible for quality“). This is a challenging problem though, as many people have built up that perception.
You make a very good point that a data loss is always bad. I’d say the time frame *can* matter (especially if you increase the timeframe, for example, a single data loss after six months might be perceived as better than a data loss after the first hour). I’ve seen “defects per line of code” used as a metric, but it means that the code is measured as being worse over time, as new defects are found, even though the actual code quality is hopefully going up. I think that this could be a useful metric though, as could “mean time between defects” – the important thing is never to use one metric in isolation. As you say, it all comes down to presenting risk/confidence to the product owner, and then making a judgement call.
January 20th, 2010 at 11:44
For me, recognizing the distinction between testing and checking has helped me see things in a very different way. I think the two are often confused and the burden of checking frequently reduces the time available for testing. Ian’s recent post on Targeted Testing proposes a way to narrow the set of all checking thus providing more time for testing.
The thoughts on the relationship between testers and developers is really interesting. At one extreme we could have developers using a test army to laboriously check every lines of code, and that he other extreme we Test as the guardians of quality poking Development with a stick. I don’t have an answer I’m afraid, but wonder if my role as tester is more about identifying areas for investment.
January 22nd, 2010 at 13:31
Michael, Arthur, Richard, thanks for your comments and feedback.
I think we all agree this is very much an art rather than a science. We need testers skilled in finding potential problems, understanding their impacts and understanding how to articulate the impact with the right people on the project. None of these things is easy, none of it boils down to a simple metric.
As people who are passionate about testing and improving the quality of the products we work on its now our job to make sure the best people think testing is the best place to be.
Jon