Just been having a chat with a new grad who is working in one of our System Test teams.
We were talking about load, stress and performance testing and I was trying to articulate the reasons behind having a workload with peaks and troughs during the day (in the same way that a customer would) rather than our usual approach to load which is run at a constant high level for a long period. However, I was struggling with a justification as to why this might be a good idea and how it would find a different class of defects.

Then I remembered the ‘Top Gear’ episode when Hammond was trying to drive a Formula 1 car around a track. Apparently F1 cars are designed to go flat out – all the time – and are pretty good at this. What Hammond discovered to his horror was that when you don’t drive them flat out then you can get into all kinds of trouble.  Cornering was a particular problem. Flat out the tyres are warm and stick to the track, the brakes responsive and the down force of the car helps it round. Anything less than flat out and as Hammond found you are all over the place.

So in this case if your system is the car and you only use Lewis Hamilton to test it then there are a lot of defects that will get missed. Fortunately few people drive F1 cars slowly on the road, but this would be a severe problem if the system under test was something like a mini.

So moral of the day, remember for every Lewis Hamilton tester you have you need at least one Hammond.

Agile is a hot topic and is having a major influence on how we all think about software, both as developers and as consumers. The idea of capability arriving in huge waves is set to become a thing of the past, as we adopt techniques to stream capability in line with consumer expectation. I see this transition in two parts:

• Embrace new software delivery models.
  From requirements and user stories to iteration commitments and user deliverables.
• Retain flexibility as a core software design concern.
  Smart design is essential to ensure there is an architectural framework to support iterative delivery.

So what does this mean? Well, we know a lot about these things independently, the exciting part is in putting the concepts together. This got me thinking; we’ve spoken a lot about testing within an agile delivery model, but maybe we should consider testing to determine how Agile the deliverable is. One approach is for the SVT team to put in a request for an additional user story, or story variation, and measure the resulting impact in some way. For example, number of additional story points, estimated delivery date, or impact to the backlog etc.

(more…)

Back in May I posted these thoughts on Agile SVT and I’m please to find I wasn’t too far of the mark all those months ago.
I was going through a presentation on ‘User Stories’ and one of the slides (page 73 out of 89) was dedicated to ‘Writing Stories for SVT scenarios’. Here’s the gist of it:

Waterfall development has a drawback – it can be so functionally driven that testing of more abstract concepts such as “is the user interface any good?”, “can I migrate from version X to version Y” and “does my software handle failures?” can suffer. Why is this?

The core reason is that these concepts often span so many functional items, that they cannot be easily contained within testing of a single functional item. They tend to slot into the system verification test cycle – yet in the waterfall method this is at the end of the development cycle and probably subject to the tightest time constraints, when it is probably too late to fix any major architectural problems.

How can agile help here? I would like to see specific stories – and even specific sprints – designed entirely around these more nebulous concepts. This necessitates a “hold the front page” attitude: a willingness to stop development of new functionality, and instead focus on consumability/failover/documentation based on testers’ experience of earlier stories. Perhaps this is a way to implement SVT in agile…

Not much activity on here recently, must be stuff going on somewhere.

Anyway, I’ve been pondering the role of testing in an agile environment more and more recently as its something we need to get right. I’ve already touched on tester driven design in an earlier post.  Now I’d like to look at three basic concepts that may help us move on. The three concepts are:

1. Forget about test phases and replace with ‘user stories’
2. The coding, testing continuum for a user story
3. The roles in an agile team

The first concept is probably the most scary. Most teams are familiar with the Design, Code, Unit Test, Functionally verify, System verify approach to developing software. So the assumption is that in an agile environment you will still need an FV phase followed by an SV phase. This is what I want to challenge.

I’d like us to think of it more simplistically. We write some code, we do some testing. What code and testing we do should be clearly defined by the user stories that are driving the delivery. So, if for example we are delivering a new API, we would have user stories to describe how the API would be used in the single user environment (equivalent to FV type testing) and stories that describe how it might be used in a high load environment (equivalent to SV type testing). So the right type of testing still gets done, we are just driven by the user story deliveries rather than by the traditional test phases.

This then leads nicely into the second concept which I’ll cal the ‘code/test continuum’. When you have user stories as described above you’ll find that some have a large amount of coding and a relatively small amount of testing. (Delivering an implementation of an industry standard API that already has a conformance test suite would be a good example of this). At the same time you will also have user stories that have very little coding, but large amounts of testing required (using the API in a complex environment under heavy load for instance).

So the two extremes may be expressed like this.

CODE………test..

code….TEST…….

But in reality the user story can fit anywhere on the continuum.

This is an important step to realize, because I think traditionally use cases and user stories have been geared towards the functional or code end of the spectrum and we need to ensure that all aspects of a delivery are considered.

Now how do we deliver on this sort of continuum? Well that’s where my third concept comes in. The roles of the agile team. For too long we have had the split in roles between coders and testers. This has to change if we are to be more agile. What we need is skilled software engineers who are capable of coding and testing. If we had a one person project then  it is obvious that that one person would need to perform both roles. so why is it that when our project team s grow  we feel we have to split the roles across different people?

If you take an ideal sized project team (about 10) I’d expect to see a spectrum of people ranging from a small number of pure coders at one end to a small number of pure testers at the other, but with the bulk of the team sitting in the middle as ‘software engineers’ who can switch roles depending on the needs of the user story. The team would morph with each delivery making much more effective use of the team’s resources.

I’ll be carrying on this thread as its close to my heart at the moment. Let me know your thoughts.

I’ve just returned from a two day ‘Disciplined Agile Development Workshop’. Inspiring stuff! It HAS to be the way to go. Short iterations delivering real stakeholder value. Test driven development, continual integration – all that good stuff. But I must admit I have been wrestling with what that means for system test (SVT). My initial reaction was, well that’s easy, everyone else in the development team moves to the agile model and then SVT use the last iteration, when the code has stabilized and all the dependencies have been sorted out, to execute their scenarios, giving the testers several iterations of wonderful ‘prep’ time – much as we did in the waterfall model.

Fortunately I have seen the light!
(more…)