“The executive who decides whether or not you will be allowed to release your product, will not be looking at your product code to make that decision. They will be looking at your defect numbers and your test pass rates.”
This is how I started a conference call last week, as part of a session to encourage more focus on test.
In a truly Agile team there are no dedicated testers or developers. Everyone is a software engineer and everyone shares responsibility for the quality of what is produced.
In an ideal world everyone would harbour a deep desire to ensure testing has been done fully, before even considering the possibility of coding more features. Sadly, in reality it is all too tempting to charge ahead with the next feature on the list. Because it is so hard to know when you are ‘done’ testing and so easy to see the features you’ve not started yet.

So I find myself trying to raise the profile of the business of testing, why we do it, and how our process ties into it. Software engineers find it easy to understand all the features that need to be written to meet the requirements. What is needed is a gentle reminder that the gateway to release is controlled by someone who will not try the product, they will not read the code, or even the documentation. The measures that they use to determine fitness for release are defect trends and test passes. They look to see that the rate of high severity defects being raised has dropped off, and that the tests are, if not 100% passing, then very close to it.

In a traditional team, one or two key people needed to worry about that and direct their fixed teams accordingly.  Part of being an Agile team is making sure that everyone understands these metrics and takes their share of responsibility to prioritise test in their planning.

Shortly after my conference call, one of the local component leads told me that another team had let him know they would not be developing the feature he wanted until the next sprint, because their plan was full with testing in the current sprint.  What’s more, he understood and accepted that this took priority over his request. So, for the moment at least, the message is getting through.

I’ve had a really exciting couple of weeks working with one of our System Test teams to define a better way of measuring test progress and product quality. For too long I’ve been fed up with the traditional test tracking metrics where we measure passes and fails or effort remaining. Historically, these measures seem to be used just because they are simple to gather. The assumption being that all you have to do is define what test cases need to be run, then track them until they all pass. The two major flaws in this are, firstly, that it’s a big assumption that the original test plan contains everything it needs to, and secondly, it is rare for any test plan to execute smoothly and at some stage in the project the project manager realises that the pass and fails aren’t telling them anything and start asking questions like “Just tell me what works and what doesn’t”. Invariably this is either impossible to determine or requires a lot of effort from the test team. At which point the simple solution is ‘Test team, work harder!’

I’ve failed miserably so far at trying to convince project teams that they should be looking at the outstanding risk in a project, rather than test case results. But I think I have finally realised why. People don’t like talking about risk. It sounds like something bad and most project teams don’t want to be associated with something bad.

The breakthrough we had this week came when my colleague Russell Finn came up with the idea of measuring the ‘confidence’ we have in the product or system rather than the outstanding risk. Now you could argue that confidence is just the inverse or risk in this case, but I think it has a much more positive spin on it.

We had been challenged by our lead engineer, Brian Cope, to redefine how we represented our status and with the help of system test leaders, Eileen Dreyer and Chris Osbourn we set about rethinking everything we do in terms of status reporting.

What we decided to show was effectively two columns of data. One showing areas of the product that we had high confidence in and one showing the backlog of areas we currently have low confidence in (or if you like the risky areas). Now, from a very simplistic view, we can answer the question ‘What works and what doesn’t?’ or at least have a good stab at it.

The next step was to work out a way of quantifying the ‘confidence’. Fortunately, this was relatively simple as we piggybacked on a piece of work that Russell had already done, where he had defined a ‘taxonomy’ for the system under test. This taxonomy split the system into its important parts, from a capability view point. With this taxonomy we were able to prioritise and apply relative weightings for each area using ‘Planning Poker’ (http://www.planningpoker.com/). A quick Friday afternoon game involving Jon Isaac, Russell, Brian and I and we had a pretty good view of the system with each area given a number of ‘story points’. (we have since done a sanity check with other members of our department and so far our estimates are holding up).

We could then chart the confidence in the system using a couple of pictures. The first showing the confidence in different areas (and their relative weighting), the second, showing the overall system. We decided to add a third ‘state’ to show areas of risk that we would be mitigating in the current iteration.

N.B The data shown here is for a fictitious system, but imagine that it is a system that is highly valued for it ability to recover from failures and outages and has a high expectation on performance.

Once we have this picture we can view automated test cases as tools that help us build our confidence in the system. Other tools included ‘manual testing’, ad-hoc testing, code reviews, code coverage metrics and ‘tester gut feel’. These other tools are not used in traditional tracking and can be a valuable source of information for determining the quality of the product. If these things feel a bit hokey now then spend a second or two thinking about what a traditional test status showing 54% pass actually means.

Riding on the back of another piece of work, where all the existing test cases had been ‘tagged’ to show which areas of the taxonomy they exercised. We held a review with the test team to weight each test case by area. Note that a test case can cover more than one area and would be weighted independently for each area. (For instance a test case might be very highly rated in the recovery area, but do a small amount of connectivity, this would mean that the test case would be weighted in both areas appropriately).

The following charts show the quality of the system during the early iterations. The highly weighted (and therefore most important areas) are being mitigated first (in true agile fashion) and we can see that a portion of recovery is now showing high confidence, a portion is being mitigated in this iteration and the rest is still outstanding in the backlog. Clearly the system is not suitable for shipping at this point.

As the iterations proceed we can see the backlog reduce and the confidence rise.

Finally we reach the last iteration and a decision must be made on whether we can ship or not. It looks like we have a small amount of risk in recovery, performance load and stress and a high confidence in everything else.

So, do we ship it or not?
The decision is still a tough one, but I’m sure that this sort of information will be far more useful than the traditional method where at this point we would be claiming 98% attempted and 94% successful!

I think this is a radical new way of thinking about product quality and will make a huge difference in how we do business.

I’d appreciate any thoughts and ideas on how this could be improved.

I’ve just been reading Mike Cohn’s ‘User Stories Applied’ for agile software development.  Excellent book – well worth a read and it’s helped me clarify my understanding of what user stories are, how to use them and how they relate to use cases.

I’ve pulled out some sound-bites that I liked:

“Story cards are the reminder to talk!”

“Just because users have the problem does not mean they are uniquely qualified to propose its solution”

Also liked his analogy for gathering requirements, where he suggests its like fishing with nets. We start with a net with big holes so we only catch the really big requirements, then over time the holes in the net get smaller. Traditional requirement gathering uses a very fine mesh net that attempts to catch everything first time round. Usually sinking the boat (I made up that bit – hope you don’t mind Mike).

I’d recommend this book to anyone involved in agile projects. There’s some good stuff in there.

Now I know you’re going to say ‘Jon, you haven’t mentioned anything about test in this post!’. Well don’t panic, I’m going to do a seperate post on my thoughts around how it releates to testing, especially system test.

How often have you heard, “We have to fix the low severity defects now, or they will never get fixed!” – even when you know some high priority core functionality is still not working?

Stepping back, fixing low severity defects may not seem in line with the agile approach, when your backlog is tackled in priority order.  However, I like to think of defects as like droplets of oil: they aggregate, and collectively they can form a much bigger droplet and a more serious problem. This seems particularly true of consumability defects.

In this case, resolving low severity defects collectively is indeed a high priority task. But there are challenges:

• How can we track this aggregation – and hence prioritise defect resolution? Informally, through impact on user stories? Or through modifying defect tracking tools to recognise that defects are not necessarily isolated issues, but instead can mutually reinforce?
• Is it an “all or nothing” approach to resolving the aggregated defects, or should you pick off individual ones to shunt the overall problem back down the priority stack?
• Finally, I have interchanged “severity” and “priority” in this post. Purists will argue these are separate attributes for a defect – but I think this post demonstrates how they can be strongly correlated.

Agile is a hot topic and is having a major influence on how we all think about software, both as developers and as consumers. The idea of capability arriving in huge waves is set to become a thing of the past, as we adopt techniques to stream capability in line with consumer expectation. I see this transition in two parts:

• Embrace new software delivery models.
  From requirements and user stories to iteration commitments and user deliverables.
• Retain flexibility as a core software design concern.
  Smart design is essential to ensure there is an architectural framework to support iterative delivery.

So what does this mean? Well, we know a lot about these things independently, the exciting part is in putting the concepts together. This got me thinking; we’ve spoken a lot about testing within an agile delivery model, but maybe we should consider testing to determine how Agile the deliverable is. One approach is for the SVT team to put in a request for an additional user story, or story variation, and measure the resulting impact in some way. For example, number of additional story points, estimated delivery date, or impact to the backlog etc.

(more…)

Myself and a colleague visited the University of Southampton’s School of Electronics and Computer Science today. We were invited as guest lecturers to deliver a presentation on Enterprise Software Testing to a group of Software Engineering MSc Students. I was really impressed with the questions and discussion points from the students. There was a great deal of interest in the testing discipline with real enthusiasm for Agile practices, as a strategy to deliver capability, and not just code.

Maybe testing really is the new development!

I have just been sent a link to a company called UTest who are building a software testing community.

Their basic model is to build a community of networked testers, invite development projects to submit their projects and then match testers with projects. Simple.

I can see this working very effectively for small scale, web based projects as it brings incredible diversity to the testing. I wonder how we could benefit from it in Enterprise testing?

Back in May I posted these thoughts on Agile SVT and I’m please to find I wasn’t too far of the mark all those months ago.
I was going through a presentation on ‘User Stories’ and one of the slides (page 73 out of 89) was dedicated to ‘Writing Stories for SVT scenarios’. Here’s the gist of it:

Waterfall development has a drawback – it can be so functionally driven that testing of more abstract concepts such as “is the user interface any good?”, “can I migrate from version X to version Y” and “does my software handle failures?” can suffer. Why is this?

The core reason is that these concepts often span so many functional items, that they cannot be easily contained within testing of a single functional item. They tend to slot into the system verification test cycle – yet in the waterfall method this is at the end of the development cycle and probably subject to the tightest time constraints, when it is probably too late to fix any major architectural problems.

How can agile help here? I would like to see specific stories – and even specific sprints – designed entirely around these more nebulous concepts. This necessitates a “hold the front page” attitude: a willingness to stop development of new functionality, and instead focus on consumability/failover/documentation based on testers’ experience of earlier stories. Perhaps this is a way to implement SVT in agile…

Not much activity on here recently, must be stuff going on somewhere.

Anyway, I’ve been pondering the role of testing in an agile environment more and more recently as its something we need to get right. I’ve already touched on tester driven design in an earlier post.  Now I’d like to look at three basic concepts that may help us move on. The three concepts are:

1. Forget about test phases and replace with ‘user stories’
2. The coding, testing continuum for a user story
3. The roles in an agile team

The first concept is probably the most scary. Most teams are familiar with the Design, Code, Unit Test, Functionally verify, System verify approach to developing software. So the assumption is that in an agile environment you will still need an FV phase followed by an SV phase. This is what I want to challenge.

I’d like us to think of it more simplistically. We write some code, we do some testing. What code and testing we do should be clearly defined by the user stories that are driving the delivery. So, if for example we are delivering a new API, we would have user stories to describe how the API would be used in the single user environment (equivalent to FV type testing) and stories that describe how it might be used in a high load environment (equivalent to SV type testing). So the right type of testing still gets done, we are just driven by the user story deliveries rather than by the traditional test phases.

This then leads nicely into the second concept which I’ll cal the ‘code/test continuum’. When you have user stories as described above you’ll find that some have a large amount of coding and a relatively small amount of testing. (Delivering an implementation of an industry standard API that already has a conformance test suite would be a good example of this). At the same time you will also have user stories that have very little coding, but large amounts of testing required (using the API in a complex environment under heavy load for instance).

So the two extremes may be expressed like this.

CODE………test..

code….TEST…….

But in reality the user story can fit anywhere on the continuum.

This is an important step to realize, because I think traditionally use cases and user stories have been geared towards the functional or code end of the spectrum and we need to ensure that all aspects of a delivery are considered.

Now how do we deliver on this sort of continuum? Well that’s where my third concept comes in. The roles of the agile team. For too long we have had the split in roles between coders and testers. This has to change if we are to be more agile. What we need is skilled software engineers who are capable of coding and testing. If we had a one person project then  it is obvious that that one person would need to perform both roles. so why is it that when our project team s grow  we feel we have to split the roles across different people?

If you take an ideal sized project team (about 10) I’d expect to see a spectrum of people ranging from a small number of pure coders at one end to a small number of pure testers at the other, but with the bulk of the team sitting in the middle as ’software engineers’ who can switch roles depending on the needs of the user story. The team would morph with each delivery making much more effective use of the team’s resources.

I’ll be carrying on this thread as its close to my heart at the moment. Let me know your thoughts.