The final production build of the software is ready. The last full run of testing starts. After two weeks, a potentially serious problem is found and the product manager decides it has to be fixed. The fix is rushed in, unit tested and built into a new production build.

Now, there are only two days left before the product must be shipped to customers. The three week final test phase now goes “out the window” and the test team do the best they can to verify that the product still works without any serious regressions.

This is an example of “Targeted Testing” being applied. Every test team has to do it at some stage, there simply isn’t enough time and resource to do an ideal job.  The effectiveness of that final test of the shipped product depends on the skills of all those involved in the decisions affecting every aspect of the life cycle of that last change – and probably a great deal of luck.

“What I need is a list of specific unknown problems we will encounter.”
(“Dilbertism” from Lykes Lines Shipping)

Time constraints like this are actually happening throughout the whole product development cycle. All the tests in the official plan may be run in several test phases, but by the end of each phase, the product has moved on, and in an ideal world, all those tests (and some not even dreamed of yet) need to be re-run.

Wouldn’t it be great if there were some tools and techniques to help do Targeted Testing so that at any time in the lifecycle of the product, we could know exactly which test is most likely to find a possible problem? All the available tests would be dynamically ordered based on this ever changing likelihood of finding problems. That way, we could be sure that the best possible testing was being done at any point in time in whatever time is available.

(more…)

I found this article in a BCS security news letter that I received in my inbox this morning.

It provides an interesting angle on the testing (checking) of web applications, as even a seemingly trivial deployment may be exploited as part of a system attack.

(more…)

Had a great meeting this morning with a number of senior testers from Hursley.
The main theme of the discussion was around an article found by Russell Finn, ‘Testing v Checking‘ – where basically checking is doing what you are told and testing is doing what you feel should be done (verification v validation might be a better way of putting it).
We have evidence from one of our major products that 50% of our field reported problems are things no one had considered (not designed, coded or tested for). So whilst we are pretty good at the verification side of things, there’s a lot we miss becuase we are not testing to real stakeholder needs.
A solution we discussed was around exploratory testing where the emphasis is on learning about a system and improving it through testing.

We also discussed quality metrics and how they scale. Traditional methods of counting defects per 1000 lines of code are fundamentally flawed because they don’t consider the impact on the end user.  One defect in a 100 line application might sound good, but what if that code is distributed across a million devices and each one needs an update?

Perhaps a better method would be ‘mean time to defect’ – some sort of measure around the time it takes an average end user to encounter a new defect.

Final thoughts were around the combative nature of a tester raising defect. Each time we raise a defect we are effectively telling the coder their baby is ugly. Is there a better way of doing this to get testers and coders working cooperating more? One suggestion was to change the term defect to ‘opportunities for improvement’ – not ideal I know but I like the sentiment. Comment if you have any better ideas.

Then try softwaretestingclub.com.

Rosie Sherry from softwaretesingclub.com left a comment to my last post after discovering TestingBlues for the first time. I’ve had a browse through the content over there and am really impressed, I particularly like the exchange – a place to ask testing questions.

Who knows, we may be able to collaborate in the future.

We recently added polls to TestingBlues, so of course I immediately went to exercise my voting right – only to be met with a badly rendered and ugly poll sitting at the bottom of the page.

Viewing a TestingBlues poll in IE6

After a bit of digging, I found out the fault’s mine: I use Internet Explorer 6. The polling feature – provided as a 3rd party plugin – doesn’t support IE6 properly. However, we have decided to keep the polling functionality for now as we believe polling functionality for 90% of users is better than no functionality at all.

In other words, we have tested this website, but made the decision to ship with a known problem. We hope readers now understand why – and aren’t under the perception that we don’t test properly. After all, that would be some irony…

Postscript: It’s my choice to use IE6 of course. It’s important to use (and test against!) older and sometimes unpopular versions of software – simply because some users may not have the luxury of the choice of a newer, shinier release. After all, there’s a lot of IE6 users out there: check out some recent usage charts on http://www.betanews.com/article/Statistics-Firefox-35-surpassed-IE7-in-global-usage-share-last-week/1261428919. This did make me think about browser usage data. Perhaps my own usage of IE 6 perturbs these metrics, so ultimately I’m testing software purely because I use it. Sort of Heisenberg’s Uncertainty Principle applied to software: an uncomfortable thought somewhat mitigated by the vanishingly small impact I have…

OK, so we’ve slacked off a bit in the second half of 2009. However, this is all going to change in 2010 – well, that’s what we told each other when we met up this week. Fresh new content and more interaction was one of the proposals, but we need your help to get it right!

What would you like to see more, or less, of on TestingBlues.com this year?

Please leave a comment to this post or use the new  poll widget we’ve added to the right column of the page.

Happy new year!

Just seen this article by Tim Rily, Director of QA at Mozilla. Negotiating for Quality Software. I very much like the pragmatic approach. We talk a lot about how testing fits into the agile way of delivering software, this article encapsulates how it can be done. The key is that everything is negotiable, engage your stakeholders to find out what is really important and use that to guide your testing.