Your test is not a beautiful and unique snowflake
I was hacking together a piece of Ruby code today. The code was only a couple of hundred lines long, but I must have tested it 100 times. The test was as much of a hack as the code. Code, test, code, test, code, test. Fixed something? Test it. Added a (tiny) feature? Change the test. Why is it then, that as part of a non-hacking team of uber-professionals, it might it be *days* before the code gets tested properly? In a waterfall process, it might be weeks.
Is it because when we make it someone’s sole job to do the testing, that the test itself becomes somehow “precious”, and not the throw-away thing that might be more useful? Or is it just a result of working in a team and not on your own?
Tags: hacking
December 2nd, 2008 at 09:30
Nicely articulated Arthur, I think this is the beauty of agile – the overall delivery is greater than the sum of its parts. If something needs to be done, then do it, if not, don’t. The waterfall method very much allowed tasks to fill the time available (and more)!
December 2nd, 2008 at 17:00
Jon has previously referred to this behaviour as the ‘test security blanket’ – Once a test has been written it can become very difficult to take it out of service without causing someone to worry. At the same time, if a given team could only afford 100 tests, it is likely there would always be just the right 100 tests.
Would this increase the risk of regression?
What if for every 100 working tests developed, there were 10 regression tests delivered?
As the regression capability grows will it become necessary to manage this set?
We have to cope with level of complexity in our daily lives. If you were feeling unwell, would you expect a Doctor to order that every known test be carried out before they make an assessment? This could take some time!
Is it more practical to work with the experts to narrow the scope?
December 2nd, 2008 at 23:48
“Why is it then, that as part of a non-hacking team of uber-professionals, it might it be *days* before the code gets tested properly?”
Depends what do you mean by “properly”? It can be agile to advocate that code be substantially tested by the person who wrote it. As in Extreme Programming, where Unit Testing by the developer is followed by Acceptance Testing done by the customer. In that case, the developer is responsible for testing his or her code “properly”. I’ve always thought it a bad idea for someone else other than the writer of the code to be responsible for the majority of the testing. That leads to the loss of a feedback loop where code writers learn their trade and a potential misunderstanding about who is responsible for code quality (a clue: it’s not the tester).
I think if you test your own code, your motivation is fundamentally different from that of people whose job it is specifically to test. Testing your own code is defensive: you want to try to minimize the number of defects that other people find because it reflects on you. Testing other people’s code is more attacking: finding as many defects as you can so that customers find fewer.
Testing your own code is more efficient than testing someone else’s. To test someone else’s, you need to understand it and the function it performs first. That is an extra overhead. You do need the sanity check of other people testing your code, because they will try things that you didn’t think of. But that should happen only once you are satisfied, through your own testing, that it performs satisfactorily.
“Is it because when we make it someone’s sole job to do the testing, that the test itself becomes somehow “precious”, and not the throw-away thing that might be more useful?”
I don’t think this has any connection with the first question of why code isn’t tested sooner. If someone’s sole job is testing, you can bet they don’t just have to test *your* code. So they also have some other code to test when yours arrives in their inbox. Yours takes its turn in their schedule.
And what about regression testing? Don’t you need that test to be long-lived as well, for future automated regression testing? You don’t want that test to be throw-away. It takes longer to prepare and correct fully automated tests.
People are reluctant to throw away tests, because much of the traditional test process is based around caution, the minimization of risk (to the project management, not necessarily the product): we did it this way before, so we’ll do it the same way again, and then no-one can blame us if things go wrong. If we throw away a test, we might get blamed for that customer problem. The way around this is to take measurements so that decisions can be taken on the basis of better information than “number of tests”. But you have to convince the project managers that the information you have is indeed better.
December 3rd, 2008 at 12:46
Some excellent points, Ian. I think the point I was trying to make about “precious” tests applies to developer-written tests as well. Once you make the test a deliverable in itself, then you imply that the quality of test is a reflection on the tester (in much the same way that you’ve pointed out that the quality of the code is a reflection on the developer). I think that maybe this should be the case for regression tests, which are “keepers”, but maybe we need to be clearer that not all tests fall into that category. Getting the balance right between the tests we keep and the ones we throw away is the tricky part.
I think you’re right about the risk associated with throwing away regression tests. I guess that having quantitive measurements such as code coverage, and being able to demonstrate fulfillment of user stories, can help in this regard.
I don’t think I’d go as far as to say that testers aren’t responsible for code quality (in the broad sense of the term). I think that responsibility should be shared by the entire delivery team (See “Who is responsible for quality?“). Given that, I agree that there is a lot of useful testing that can/should be done by the developers. I think that by breaking down the separation of responsibilities of developers and testers, and instead having unified delivery teams, we can help ensure that we’re not (implicitly) measuring the quality of the tests themselves, but the quality of what the team as a whole delivers.