When do your testers join the software development process? If you're like most teams, testing doesn't start until late in the development cycle. Once testing is engaged, you work around the clock finding the important bugs, fixing those you can fix, and trying to get the code ready to ship. In all, your testers and developers will put in several extremely late nights and probably a few weekends to get the code out the door.  
 
Last-Minute Heroes 
Testing that starts late in the software development cycle gives many bugs that were originally small and simple a chance to become deeply entangled with other bugs in the system. These entrenched bugs are often difficult to find and nearly impossible to fix well.  
 
On the bright side, everyone who survives the dreaded days of debugging will likely be proclaimed a hero. But I wonder if heroism in these situations is as necessary as we are often led to believe.  
 
Let's step back for a moment and look at similar heroics in a totally different context. Smokejumpers are firefighters who parachute into forest fires armed with shovels, picks, and little else. This matches up pretty well with my idea of heroism. Their work is important, exhausting, and dangerous. Fatalities occur on a regular basis. 
 
But an ounce of prevention may really be worth a pound of cure--or hundreds of acres of forest. Some communities have begun using goats to deal with wildfires. Instead of waiting until a fire starts, these communities hire goats to munch away on the underbrush. This starves a potential fire of the fuel that would allow the flames to spread. This is not glamorous work, but the goats effectively clear many acres in a short time. 
 

Mike McMillan/Spotfire Images

Goats vs. Smokejumpers 
1. Goats work cheap. 650 goats probably cost as much as one firefighter.  
2. Goats are not nearly as exciting as smokejumpers. Watching goats munch on underbrush is only marginally more interesting than watching the underbrush grow.  
3. Using goats requires planning. Goats are of little use once the fire has started. (Unless you plan to feed them to the firefighters.) To be effective, they must be used before the emergency hits.  
4. Goats are not a complete solution. Fires will still happen and you will still need firefighters—hopefully not as many or for as long.  
5. Measuring the success of goats is tricky. You can't talk about fires extinguished; you have to talk about fires prevented. Fortunately, if you have some idea of how many fires would usually flare up, you can get an estimate of how helpful the goats have been. 
 
Now, let's return to your team's software development efforts. On your last project, did you get pulled into firefighting mode because your testing found serious bugs only late in the development cycle? Did you repeatedly send in your developers to contain and debug the problems? I'll bet your developers worked mainly in reactive mode, moving from one hot spot to the next since they had no time to create long-lasting solutions. In other words, your people were heroes heading straight into burnout. 
 
But what if your team had taken a more goat-like approach, stopping small, isolated bugs before they could become large, tangled bugs? What if your developers and testers used cheap, simple methods during the development phase to lessen the presence of bugs later in the process? 
 

Reused with permission: San Diego Union-Tribune

The truth is we have had such tools and methods around for a long time. We just haven't always used them as well or as often as we could have. For instance, to name just a few: 
 
1. Code reviews and inspections. Code review is probably not on anyone's list of exciting ways to spend a few hours, but its reported 70-80% find rate certainly deserves attention. 
2. Automated Inspections. For the lazier among us, there are tools that will automatically inspect code for suspicious constructs.  
3. Unit testing. With the emergence of extreme programming, unit testing is now in fashion.  
 
Early Detection vs. Late Night Heroism 
1. Early detection tools cost much less than finding showstopper bugs at the last minute. 
2. Early detection methods find bugs in a simple and boring way, which is good. Software development currently has too many nail-biting moments and death marches. 
3. Early detection tools require planning. Once you are in reactive mode, it's too late. 
4. These tools do not find all bugs. You still need testers and developers to fix bugs—hopefully not as many since the number of bugs will be smaller. 
5. Measuring the impact of these tools can be tricky since their real value is in preventing worse bugs down the line. 
 
Keeping It All Under Control 
Being a hero is great, but only if your heroism doesn't involve a situation you could have avoided in the first place. Good engineering doesn't consist of random acts of heroism. Good engineering should celebrate a subtler form of accomplishment. 
 
Further Reading 
1. For quick background on the effectiveness of code reviews, read "Inspections – Evolution and History: 1972-2001" by Michael Fagan at http://www.sdm.de/download/sdm-konf2001/f_7_fagan.pdf  
2. Jasper Kamperman's article on Automated Software Inspection, at www.stickyminds.com/se/S3639.asp, provides a good overview of how Lint-like tools can improve productivity. 
3. www.JUnit.org and www.NUnit.org provide good starting points for finding out about the latest craze in unit testing. 
4. For astounding photos of smokejumpers in action, visit Mike McMillan's website, http://www.spotfireimages.com.  
5. For un-astounding photos of goats in action, visit http://www.azcentral.com/specials/special38/articles/0701goats01.html and http://www.signonsandiego.com/uniontrib/20040416/news_1n16goats.html  
6. And, finally, "Nobody Ever Gets Credit for Fixing Problems that Never Happened," at http://web.mit.edu/nelsonr/www/Repenning=Sterman_CMR_su01_.pdf is a good read about making sure that prevention techniques get recognized, rewarded, and sustained.

About the Author
Harry Robinson is Test Architect for Microsoft's Engineering Excellence Group, where he works with product teams across the company to identify and promote advanced test technologies. He writes and speaks frequently on software testing, and for the past six years he has been a driving force behind Microsoft's model-based testing initiative. You can reach him at harryr@microsoft.com.

Back to Top 
 

	Comment:     by Michael Cummings 10/15/2004 Great article. It was well-received by both the test and dev teams here. One comment I would make is that we have to try to make sure that the heros are not the ones starting the fires - which seems to be the case all to often. MC Author's Response: 10/15/2004     You make an excellent point. According to Fire Chief Thomas W. Aurnhammer (http://nmfirechiefs.com/article-aurnhammer1.html), firefighter arson is “a crisis [that] needs to be acknowledged and addressed.” Aurnhammer writes: “The fire fighter arsonist could be someone looking for recognition or who likes to play the role of the hero. ... ... Some have also noted the social need of wanting to be accepted by others, or wanting to belong to or be identified with a significant group. Praise and being recognized by peers also comes into play. ... the motivation is usually not intended to be harmful. The excitement of fire fighting combined with a maturity level that does not realize the harmful effects or the severity of the crime being committed may also be another factor.” I wonder how much this might apply to software heroes and crises? Regards,
	Comment:     by Clay G 10/13/2004 I say the Best time for catching "bugs" is before they are even bugs. I think the bugs you find by reading through code are generally the "you didn't handle null" type, while I mainly see really serious and expensive bugs from misunderstood requirements and incomplete business rules. They are not usually detectable by reading code because they're about how the system works as a whole. Your "goat" was okay, but from from the title I thought it might be about acting stubborn as a goat (bad rap?) in not allowing shoddy work or unanswered questions to slide past, requiring heroics later on. Because there is usually a pressure to be rushing...Read On Author's Response: 10/14/2004     Thanks for your comments. Early detection doesn’t need to be limited to code checking. It can also include formal approaches that check the consistency between a program and its specifications. A good explanation of this approach is in the paper “The Spec# Programming System: An Overview” at http://research.microsoft.com/~leino/papers/krml136.pdf. Regards,
	Comment:     by Steve Blesson 10/13/2004 Very interesting article, thank you. I remember my boss was telling me last week that QA positions have no exposure and visibility at all. QA analysts and testers are never heroes, only developers and project managers get all the credits for success. However, we have the obligation to detect problems early enough to keep stress levels very low in projects and make sure their outcomes are successful. One day, may be, some literate project managers and developers will recognize our true value in the process. Let's just keep doing our job which is helping produce high-quality product...:-) Author's Response: 10/14/2004     I agree with you that QA’s job is to help deliver great software, but I worry about our long-term prospects if we don’t get recognition for our goat-ish efforts in the shorter time-frame. If management doesn’t see our value, why should they keep us around? In the past, when prevention efforts have caused the bug count in system test to drop, I’ve had to get creative in characterizing tester contribution. My favorite unofficial metric was “How many people on the team are sleeping well at night?” Sleeping well was directly related to the number of crises our work had prevented, and it was a metric that everyone agreed reflected their confidence in the software. I doubt that you’ll find the “sleep well” metric in most textbooks, but it did the job for us. Regards,
	Comment:     by Srinivasan Desikan 10/13/2004 This is one of the most effective ways of saying "defect prevention is better than fixing". I can afford to keep only one hero in the company but as many goats as possible, sothat I can get along more effectively. Two heroes are difficult to manage but goats can always work together as a team. Your analogy/example is very good and I wish you write often in stickyminds. Author's Response: 10/14/2004     And sometimes even one hero is too many, especially if that hero has only ever worked in adrenaline-charged situations and has trouble trading the bleeding edge of crisis management for the “bleating edge” of just getting good software out the door. Thanks very much for your kind words on the article!
	Comment:     by Fred Earl 10/13/2004 Very funny. (I mean it.) But goats don’t prevent bridges from collapsing or airplanes falling from the skies. Goats’ big problem is that they require the entire ecology to be in place before they can get to work. Developers find that it takes more effort to create “testable” units for testers early in the process than to simply check their work themselves. Inexperienced and lazy developers don’t even do it themselves (beyond the minimally useful “unit tests.”) “Early detection” tools identify the kinds of simplistic errors that only beginners create. Software will remain only as good as its developers until the field matures and becomes as...Read On Author's Response: 10/14/2004     Thanks for your comments. (I mean it too!) You bring up great issues (even though we disagree on almost every point). Planes, bridges and goats: anytime engineers build models or check their calculations they are using “goats” because it is more effective than building bridges and watching them fall until we hit on the right approach. One way the software field will advance is to stop being in reactive mode Every approach, goat or hero, requires support from the environment. Smokejumpers would be nowhere without their planes, and last-minute software heroes would be lost without their debuggers. We just don’t think of those reactive tools as “ecology” since they are so common in how we deal with fires nowadays. It’s true that goats need their own support environment; what makes goat ecologies stand out is that their needs are different from the mainstream environment we are used to. People are fallible and even the best developers make mistakes. Early detection tools have come a long way from finding simplistic errors. Check out http://user.it.uu.se/~kostis/Seminars/PLI/software_eng.html for interesting papers on the latest happenings in the field. Best regards,
	Comment:     by stephen kay 10/13/2004 This reminds me of the story of a chinese family of healers. One of them becomes famous for his skills and is called infront of the great emporer. The emporer asks him who is the greatest healer in his family. He says that he heals the sick, sometimes with dramatic results and is hailed a hero throughout the country. His elder brother heals those whose illness has just started to take root, and he is known only in the local village. His eldest brother prevents sickness before it starts, and he is unknown outside of his own home. Author's Response: 10/14/2004     Hi Stephen – Thanks for bringing up a great analogous situation. I’m always bothered by eldest brother’s situation, though: How does he make a living if no one outside the home knows about him? Thomas Cleary mentions the story of the healers in his translation of The Art of War, and he goes on to quote the Tao-te Ching: “Plan for what is difficult while it is easy, do what is great while it is small. … For this reason sages never do what is great, and this is why they can achieve that greatness.” But then The Art of War points out the down side: “When trouble is solved before it forms, who calls that clever? When there is victory without battle, who talks about bravery?”
	Comment:     by Gene Fellner 10/12/2004 Americans as a people are problem solvers, not problem preventers. We happily rebuild homes destroyed by hurricanes, brushfires, and mudslides. In their original locations! And our banks, insurance companies, police, firefighters, and relief agencies happily subsidize this insanity! (One definition of "insanity" is: "The belief that if you keep doing what you've always done you will somehow stop getting what you've always gotten.") This archetype has been around for so long that it's got a name: The Loose Cannon Syndrome. Named after an infamous event on a British warship of the ropes-and-canvas era. As a storm approached, a young officer...Read On Author's Response: 10/14/2004     Thanks for the example and the etymology! The method seems a bit harsh as manager-employee feedback, but I suppose it got the message across. I have certainly been on projects where getting rid of the “hero” (in a nicer way than the British) have saved us miles of grief. The problem is: how do we make quiet competence more exciting and noticeable?
	Comment:     by William Gentry 10/11/2004 I wholeheartedly agree. Our Software Engineering department has turned to code reviews, automated code inspections, unit testing and even low level system testing within the last year. As a result, the number of defects found by SQA team has been dramatically reduced, which tells me that the little bugs are found and corrected before they get to us. Now if we (the company) could schedule better so that SQA has enough time to run through the test plan and prevent even more overtime and unnecessary heroism... Author's Response: 10/14/2004     It’s great to hear that you’re seeing fewer bugs. Make sure that your management understands how much your work contributed to the decline. Otherwise, they may think that the drop in bugs means that the processes are no longer necessary! Or worse, they’ll attribute the improvement to “a really top-notch bunch of developers” and distribute rewards accordingly.
	Comment:     by neill mccarthy 10/11/2004 Harry, another good column - shame about the images. Goats are wonderful creatures but do not get the best of press, a bit like QA and Testers. I have been trying to instil positive images for some time into my test team...now I have to sell them the image of being goats!!! Author's Response: 10/14/2004     Thanks, Neill. You are right - goats have a definite public relations problem. So maybe we can help re-shape the image of goathood. For instance, I think the Agile community with its emphasis on unit testing would be a natural supporter of the low-key heroism of goat-hood. After all, there are few things in the world as agile as a goat. And in case the pro-goat software development movement does take hold, Mike McMillan of Spotfire Images has already suggested a good bumper sticker motto: “Long live the brave, the proud, the goats.”
	Comment:     by Jim Bernstein 10/11/2004 Harry One of the most truly entertaining articles on Sticky Minds! The problem you describe plagues most industries however most companies are so busy firefighting they can't go to the market to buy the goats. I try to convince people each day that finding a problem in development costs much less than finding it in pre (or worse) post production. Jim Solstice Software Author's Response: 10/14/2004     Thanks for the kind words, Jim! And even if the company does make it to the market, are they more likely to buy shiny firefighting equipment or a disreputable looking bunch of goats? Which purchase would their management be more likely to approve? As an industry, we need to educate management that prevention and early detection are the right things to do, but I agree that it's not an easy sell.
	Comment:     by David Ramger 10/11/2004 I heard on ESPN recently a comment from one of the sportscasters who was discussing baseball's weekly Top 10 Plays of the Week. He pointed out that a large number of the exciting diving catches that were being highlighted on the show were a result of a very good athlete making a great play only because he was out of position. There seemed to be a lot of younger players who made the highlight reel simply because they were good athletes who *had* to make a diving reach to catch a ball that the older, more experienced players made easily because they were in the right spot in the first place, but made the same catch look routine. The...Read On Author's Response: 10/14/2004     That’s a great analogy. It’s good that the athletes are paid by the team owners, not the sportscasters! Team owners only care that the ball gets caught, not how much effort went into catching it. Sportscasters need to have some excitement if they want people to tune in. Good development should be a boring spectator sport.

Back to Top