Anti-spamming software is an integral part of many websites, often taking the form of “type in the distorted text you see”. The assumption is that this is – hopefully – too difficult for a machine to do quickly and trivially, but easy for humans with our better pattern-understanding skills.

reCAPTCHA is an example of this which takes this a step further: the second word is used to confirm I am indeed human and not spamming, while the first word is text from a failed attempt to digitize historic documents: i.e., reCAPTCHA provides a (free!) way to get humans to help digitize old text from before the computer era. This is a great idea, but does throw up some interesting examples in practice:

It’s obviously rather harsh to expect someone to be able to enter (i) a mathematical forumla with superscript and subscripts and (ii) Greek text, particularly as the text box only accepts plain English with no formatting. More worryingly, I got thinking about whether other even more inappropriate (i.e., offensive) data might wing its way to the user. I followed this up, and got the following response:

“The facility does have filters in place to prevent offensive words coming up…  some of the words in these texts are difficult for computers to process, we are using the results of your efforts to help decipher them.”

Which makes me wonder: if the texts are difficult to process, how can we truly be confident in any filters? And how should a tester go about raising a concern like this: is it a real defect? Or just a worry…?

Just been involved with the beta testing of the new Feature Pack for OSGi Applications and JPA 2.0 for IBM WebSphere Application Server V7:

OSGi and JPA 2.0 FeP

We’ve been using a much more agile development process than in the past, continually tweaking our approach and trying out new development and test tooling. I’m really pleased to say that this investment has paid off: even as a hardened test cynic, I’m genuinely impressed by the quality.

“The executive who decides whether or not you will be allowed to release your product, will not be looking at your product code to make that decision. They will be looking at your defect numbers and your test pass rates.”
This is how I started a conference call last week, as part of a session to encourage more focus on test.
In a truly Agile team there are no dedicated testers or developers. Everyone is a software engineer and everyone shares responsibility for the quality of what is produced.
In an ideal world everyone would harbour a deep desire to ensure testing has been done fully, before even considering the possibility of coding more features. Sadly, in reality it is all too tempting to charge ahead with the next feature on the list. Because it is so hard to know when you are ‘done’ testing and so easy to see the features you’ve not started yet.

So I find myself trying to raise the profile of the business of testing, why we do it, and how our process ties into it. Software engineers find it easy to understand all the features that need to be written to meet the requirements. What is needed is a gentle reminder that the gateway to release is controlled by someone who will not try the product, they will not read the code, or even the documentation. The measures that they use to determine fitness for release are defect trends and test passes. They look to see that the rate of high severity defects being raised has dropped off, and that the tests are, if not 100% passing, then very close to it.

In a traditional team, one or two key people needed to worry about that and direct their fixed teams accordingly.  Part of being an Agile team is making sure that everyone understands these metrics and takes their share of responsibility to prioritise test in their planning.

Shortly after my conference call, one of the local component leads told me that another team had let him know they would not be developing the feature he wanted until the next sprint, because their plan was full with testing in the current sprint.  What’s more, he understood and accepted that this took priority over his request. So, for the moment at least, the message is getting through.

The final production build of the software is ready. The last full run of testing starts. After two weeks, a potentially serious problem is found and the product manager decides it has to be fixed. The fix is rushed in, unit tested and built into a new production build.

Now, there are only two days left before the product must be shipped to customers. The three week final test phase now goes “out the window” and the test team do the best they can to verify that the product still works without any serious regressions.

This is an example of “Targeted Testing” being applied. Every test team has to do it at some stage, there simply isn’t enough time and resource to do an ideal job.  The effectiveness of that final test of the shipped product depends on the skills of all those involved in the decisions affecting every aspect of the life cycle of that last change – and probably a great deal of luck.

“What I need is a list of specific unknown problems we will encounter.”
(“Dilbertism” from Lykes Lines Shipping)

Time constraints like this are actually happening throughout the whole product development cycle. All the tests in the official plan may be run in several test phases, but by the end of each phase, the product has moved on, and in an ideal world, all those tests (and some not even dreamed of yet) need to be re-run.

Wouldn’t it be great if there were some tools and techniques to help do Targeted Testing so that at any time in the lifecycle of the product, we could know exactly which test is most likely to find a possible problem? All the available tests would be dynamically ordered based on this ever changing likelihood of finding problems. That way, we could be sure that the best possible testing was being done at any point in time in whatever time is available.

(more…)

At the end of 2008 I switched credit cards from provider X to provider Y, lured by the promises of untold wealth from the cashback deal they offered. However, I’m yet to have a month where using this card has gone smoothly: which makes me wonder how well their application and business logic has been tested.

It was all triggered by me setting up a direct debit to pay the card off each month, and I thought nothing more of it. I was therefore somewhat surprised when the next month no payment was taken and I was spanked for interest. I phoned up – and was informed that my direct debit had indeed been set up, but flagged to start in the year 8888. At first glance, this looks like a user error which the software should reject. But then you could argue that 8888 is a valid year – just a very unusual one to use in a direct debit.

So where do we mark the cutoff between sensible data entries and wrong ones? Can we implement fuzzy logic for such subjective testing? And if so, how can we guarantee repeatable behaviour between tests – which is critical for automated testing?

Anyway – the direct debit is now working, but this appears to have triggered me travelling down some sort of error path in the credit card company’s business logic. I’ve decided to stop using this card for 2 months to let the dust settle: and then perhaps I’ll test their business logic some more…

Over the holidays I got asked, “what makes a good software tester?”. As well as recounting the traditional traits of sheer good looks, athleticism and sophisticated banter, this actually got me thinking back to what we are here to do. Crudely speaking, we are here to find defects. [Note that I don't state "find and fix defects" - as that's a topic for another discussion].

To do this well requires a broad range of character traits – just like our customers have a broad range of approaches to using our software. So when asked if I’d want a tester who takes a very rigorous approach, or one with a much more ad-hoc attitude to testing – I’d happily take both.

Wow, it’s been a while since I last posted an entry – is that how blogging works or am I supposed to do it every day . Maybe it’s all the Olympics watching that’s been going on!

Anyhows, I was looking for some quality criteria in Testing…

(more…)