The final production build of the software is ready. The last full run of testing starts. After two weeks, a potentially serious problem is found and the product manager decides it has to be fixed. The fix is rushed in, unit tested and built into a new production build.

Now, there are only two days left before the product must be shipped to customers. The three week final test phase now goes “out the window” and the test team do the best they can to verify that the product still works without any serious regressions.

This is an example of “Targeted Testing” being applied. Every test team has to do it at some stage, there simply isn’t enough time and resource to do an ideal job.  The effectiveness of that final test of the shipped product depends on the skills of all those involved in the decisions affecting every aspect of the life cycle of that last change – and probably a great deal of luck.

“What I need is a list of specific unknown problems we will encounter.”
(“Dilbertism” from Lykes Lines Shipping)

Time constraints like this are actually happening throughout the whole product development cycle. All the tests in the official plan may be run in several test phases, but by the end of each phase, the product has moved on, and in an ideal world, all those tests (and some not even dreamed of yet) need to be re-run.

Wouldn’t it be great if there were some tools and techniques to help do Targeted Testing so that at any time in the lifecycle of the product, we could know exactly which test is most likely to find a possible problem? All the available tests would be dynamically ordered based on this ever changing likelihood of finding problems. That way, we could be sure that the best possible testing was being done at any point in time in whatever time is available.

(more…)

At the end of 2008 I switched credit cards from provider X to provider Y, lured by the promises of untold wealth from the cashback deal they offered. However, I’m yet to have a month where using this card has gone smoothly: which makes me wonder how well their application and business logic has been tested.

It was all triggered by me setting up a direct debit to pay the card off each month, and I thought nothing more of it. I was therefore somewhat surprised when the next month no payment was taken and I was spanked for interest. I phoned up – and was informed that my direct debit had indeed been set up, but flagged to start in the year 8888. At first glance, this looks like a user error which the software should reject. But then you could argue that 8888 is a valid year – just a very unusual one to use in a direct debit.

So where do we mark the cutoff between sensible data entries and wrong ones? Can we implement fuzzy logic for such subjective testing? And if so, how can we guarantee repeatable behaviour between tests – which is critical for automated testing?

Anyway – the direct debit is now working, but this appears to have triggered me travelling down some sort of error path in the credit card company’s business logic. I’ve decided to stop using this card for 2 months to let the dust settle: and then perhaps I’ll test their business logic some more…

Few software engineers would argue against the importance of automation, but the code that tests code is often hidden away and shies from the limelight. Automation tools receive extensive and repetitive use, and as a result, they are generally very well tested. Without doubt this is an innovative space, the things we test are sophisticated which demands great sophistication and creativity from a test harness.

Test automation is certainly not unique to the software industry. The things we use every day have all been tested in some way, many in an automated fashion. I always become engrossed with the Dyson testing advert when I catch it on TV. They have put their test automation tooling on centre stage and it is fascinating to watch in action.

http://www.dyson.co.uk/testing/