The final production build of the software is ready. The last full run of testing starts. After two weeks, a potentially serious problem is found and the product manager decides it has to be fixed. The fix is rushed in, unit tested and built into a new production build.

Now, there are only two days left before the product must be shipped to customers. The three week final test phase now goes “out the window” and the test team do the best they can to verify that the product still works without any serious regressions.

This is an example of “Targeted Testing” being applied. Every test team has to do it at some stage, there simply isn’t enough time and resource to do an ideal job.  The effectiveness of that final test of the shipped product depends on the skills of all those involved in the decisions affecting every aspect of the life cycle of that last change – and probably a great deal of luck.

“What I need is a list of specific unknown problems we will encounter.”
(“Dilbertism” from Lykes Lines Shipping)

Time constraints like this are actually happening throughout the whole product development cycle. All the tests in the official plan may be run in several test phases, but by the end of each phase, the product has moved on, and in an ideal world, all those tests (and some not even dreamed of yet) need to be re-run.

Wouldn’t it be great if there were some tools and techniques to help do Targeted Testing so that at any time in the lifecycle of the product, we could know exactly which test is most likely to find a possible problem? All the available tests would be dynamically ordered based on this ever changing likelihood of finding problems. That way, we could be sure that the best possible testing was being done at any point in time in whatever time is available.

Great in theory, but how could this be achieved in practice?

Well, techniques like Agile and Risk Based Testing etc. do help, especially with testing the new function. But, the real benefit of Targeted Testing is in regression testing where potentially thousands of long running tests can be reduced to just a few. A project I’ve worked on for several years used Targeted Testing, and I developed some prototype tools to help automate it. The benefits have been impressive and I’d like to share some interesting parts of the story in future posts. For now, I will stick with a quick introduction and cover some of the benefits and problems.

Targeted Testing is about knowing at any time:

• which components of the product are most at risk of having problems
• how serious (for customers) these potential problems might be
• which tests are most likely to expose these problems

Of course, it is impossible to answer these questions with 100% certainty, but when you consider all the data you may have access to about change history, test run history, past defects, test code coverage, expert knowledge etc. you may be surprised how much of this could be applied to Targeted Testing.

What we need to do is maintain a list of all the components in the product and their relationship with each of the tests. Each component will have a particular level of risk at any time. Each relationship with a test will have a level of effectiveness with which that test will mitigate the risk. We may even have a confidence factor for each test based on how reliable we feel its results are.

If our components are fairly large and we have a smallish number of tests, it might be possible to construct these relationships manually and benefit form the approach without the need for any tooling more complicated than a spreadsheet. However, to achieve the most benefit, we need to identify many small components, possibly every class, method or even branch in the code, and map each of these to every applicable test. One way to do this is to gather code coverage information when the tests are run. The proportion of component code covered by each test would give a measure of its effectiveness at mitigating the risk in the component.

We could be dealing with a lot of relationships here. In a small project with say 100 classes and 100 tests there would be a maximum of 10,000 class – test relationships, possibly 100,000 method – test relationships and probably a million branch – test relationships.  So we need to choose the granularity of our components carefully and also consider how easily we can determine their current level of risk.

Risk is usually increased when a component is changed and most code library systems provide queries that will give us this information. You can see how, with all this data, it is possible to order the tests by the total risk in all the components that they are able to mitigate.

These are the basic concepts of Targeted Testing. It can be applied at many levels and use many different techniques for gathering the data needed. When it is applied to regression testing throughout the product life cycle, you should see these benefits:

• possible problems will be detected earlier
• the test team will have more time to do valuable (and interesting, which is good for morale) exploratory or other testing activities.
• a better understanding of risk
• more builds can be tested

I am not suggesting here, that Targeted Testing is the whole solution. You will need to combine it with other techniques such as Unit Testing, but it is certainly worth considering as part of the testing mix.

There will probably be concerns voiced that you are not running all the tests all the time, but we have already established that this is impossible or you wouldn’t need Targeted Testing anyway.

Another issue is reporting. The product manager will not be seeing that S curve tracking to 100% tests passing. But you should be able to provide a better assessment of how much risk remains, which I believe is more valuable, especially when you consider that Targeted Testing reports can show which components contain high risk and have not been tested at all because no tests go anywhere near them. This is something that 100% test passing does not tell you.

In future articles, I hope to cover some more of this and give more detail. Until then, I welcome any comments, ideas or experiences you wish to share on this subject.

Tags: automation, quality, risk, techniques, testing

This entry was posted on Tuesday, January 19th, 2010 at 14:51 by Ian Holden. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.