Defect tracking systems are one of a software organization’s “family jewels.” They frequently control how work gets assigned to engineers and provide the data to support how managers decide when to ship. They also determine which defects to ship and which to defer. These are clearly business-critical decisions.
Unfortunately, senior decision-makers often seriously undervalue the problems these systems solve. So when it’s time to invest resources in a new system, they allocate the resources as an afterthought. After all, it isn’t a “real” project (the kind that generates “real” revenue for the company and ships to “real” customers). Money, time, and resources are all underestimated, commonly assuming it might take two engineers a couple of months to build the system. But unless you set these expectations correctly up front, no matter how hard you work to implement your system your chances of success will be severely limited. Better to take a broader view and treat this mission as a real project right from the start.
Understanding the Scope of the Project
Even before the project starts, you need to discover how much the new defect tracking system is really worth. Then you can get a better idea of how much time and money are reasonable to invest to get it. One way I do this is to gather some interested and important parties together, such as development managers, testers, and test managers, for a kickoff session. The very first question I ask in this session is:
“What is a highly successful solution really worth to this company?”
In many software development organizations, the defect tracking system is the primary means of communication between workers. So what’s that worth? For most IT businesses, the answer is “quite a lot.” Look at it this way: You’ll end up paying for the system one way or another. You can pay for the system by under-investing and then getting poor productivity and poor decisions for months or years to come. Or you can put up the cash (in organizational time and effort as well as money) and get a system that really supports your work. Defect tracking systems, especially for larger organizations, are expensive.
And if they work well, they’re well worth it.
Once you start this project, one of your first tasks will be to estimate the time required to put together the new system, and what the price range for new tool purchases will be. It is important to get this information into the approval process early. “Sticker shock” quickly cancels many much-needed improvement projects, when decision-makers are suddenly confronted with good systems costing hundreds of thousands, and sometimes millions, of dollars.
The bottom line is “you get what you pay for.” Problems do not shrink to fit the amount of resources you’re willing to expend to solve them. A defect tracking team must take it upon themselves to treat the project as if it is as real as any other project, and vow to deliver a product (albeit an entirely internal product) which is up to the quality standards of the rest of the company’s products. The central lesson to be learned is that the clients—the ones who are paying for the solution—need to have a clear understanding about how much the solution is really worth so they can intelligently decide what to do to solve the problem.
Work with Management to understand what funds and approvals will be required. Determine the cost in lost time to market, customer-discovered defects, and the development rework you’ll have to do without the system.
Once you determine and secure adequate funding, then you can move on to lay out the steps of the project. To get the system up and running, you’ll have to complete eight major tasks:
- Form the team
- Define requirements
- Decide fundamental design issues
- Choose the tool
- Implement the system
- Test it
- Deploy the system
- Monitor the system
Form the Team
Building and installing a corporate-wide defect tracking system takes a small but well-balanced development team. All the roles you would normally have on any other project are required here too, if you’re going to successfully build and deploy this system.
One of the most important roles your team will be relying upon is that of a strong negotiator. Your defect tracking system will require purchasing components from vendors. You will most surely be working with Upper Management for a larger budget. And at some point your defect tracking system will involve MIS, Customer Support, QA, and Development groups—all of whom have differing needs, time commitments, and agendas.
To build a successful defect tracking system you will need to include all of the system’s users in your project process in some way. No one individual has a complete view, because everyone’s needs are different. Development managers need to track progress towards their ship dates. Programmers need to determine what bugs are theirs to fix. QA managers need to determine that the testing is effective. And testers need to enter bug data both when opening the bug and later when regressing and closing it.
But there are simply too many people who will eventually use the system in your project for you to include everyone; so go for a representative sample. Find a product manager interested in the data the new system can provide. Find programmers who want a better system. And get the MIS group involved; often defect tracking systems ignore the MIS department, forgetting that this group will be providing the hardware and support to keep the system running for many years.
Most of the staff will have full-time assignments on “real” projects, and will be able to devote only part of their time to this mission. Don’t discount these busy people’s ability to contribute; the benefit of having a variety of voices on the team will make all the difference as you work towards a solution, and will mean easier acceptance at delivery.
You may encounter people volunteering to be on the team. Thank them for their willingness to help, but be cautious about building a team solely from self-selected participants. Screen for any red flags that might indicate your volunteers have an ax to grind or a personal agenda which differs radically from the team’s goal.
Define Requirements
For requirements, look at what you need—don’t just look at the products and pick the one you hope will work. Have your team list the features required by your organization. Typical requirements may call for a tool that can:
- Integrate with your support organization’s call tracking system
- Integrate with Development’s configuration management system
- Support your defect-resolution workflow
- Scale to your needs
- Support the metrics you need
- Produce reports to assist all users
- Be either distributed or centrally located
Lead your team through an examination of your critical requirements. How many people will use the system? What machines will they be using? Are they all on one LAN or are some remote? How many time zones are involved? [See Elisabeth Hendrickson’s article “Evaluating Tools” (Volume 1, Issue 1, page 38) for an overview of the questions you need to ask.]
Write down each requirement and which primary user the requirement serves. Don’t worry if you end up with lots of requirements, or if no one product meets all of the requirements—you can use this list as a checklist when evaluating products and making compromises during your design.
Decide Fundamental Design Issues
There are several important choices you have to make at this stage:
- Start with the schema and workflow you want to implement. Divide the issues into process- and tool-related problems. If, for example, defects are being reported but not getting assigned and fixed, you have a process problem.
- Determine if you need to comply with IEEE Standards 1044-1993 and 1044.1-1995 (dealing with defect classification), and if so how to apply them. Even if you don’t need to comply with the IEEE standard it can be very useful as input to your design, since it lays out most of the possibilities for the data you might wish to track. Pick which values you will use for the fields you want to track.
- Build a state transition diagram for a defect, showing all the states from found to closed or deferred. Determine who can change each state. Build the data dictionary, showing each data element and the values it can contain. Doing all this before you start evaluating solutions will give you a better shot at choosing the right one.
- One big choice in design is to make the buy-versus-build decision. There are many products to chose from, but your three basic options are:
- Build from scratch
- Buy off-the-shelf
- Buy then customize
Make the choice, invest the money, and once the money is spent be committed to your strategy.
Choose the Tool
One size doesn’t fit all. Defect tracking systems can incur initial costs ranging from nothing at all (e.g., free products such as GNU) to over a million dollars, and they are designed to meet different needs. Obviously, choosing a system for a small soon-to-go-public startup company (operating out of a one-room office) is not the same as choosing one for a large multi-national defense contractor (which might have development projects underway in several countries at once).
There is a wide range of solutions available to you. By carefully drawing up your requirements, you’ve already laid the groundwork for making a good decision; now match those requirements to the products available. Check the Web for the latest company offerings. (See this article’s WebInfolink for an extensive, up-to-date list of product offerings.) A Web search will also find companies who will support these products; many have useful white papers. Most of the commercial vendors offer evaluation copies of their products.
Defect tracking systems may be purchased in any of at least three distinct packaging options:
- Standalone, commercial off-the-shelf (COTS) products
- Integrated with one or more companion tools such as call tracking, automated testing, or configuration management products
- Bundled with other software engineering tools (usually tools from the same vendor)
Tools can also be compared by complexity and price:
- Simple turnkey tools, moderately priced, most of which are PC-based
- Complex and expensive off-the-shelf packages which require considerable customization (available in a mix of PC and UNIX systems)
- Collections of tools (often in Perl) which you must integrate with your database, with prices ranging from free to $1000 (most of which are UNIX-based)
Use your set of requirements and the design decisions you’ve made about how you’re going to manage defect information to evaluate the set of possible tools. Each different tool has both strengths and weaknesses. Get a clear picture of what those are and how they relate to your specific needs; then you can make an informed trade-off and choose the best one.
Implement the System
Your implementation may be as simple as opening the package and typing “setup” or it may take months of programming. You will probably need to perform some customization of any purchased package, even the most standalone ones. For example, every implementation will require adding the data specific for your company: people, teams, projects, sub-areas within projects, and status codes all need to be added to the database before it can be used. The data from one or more current projects will need to be imported.
The deployment of a new defect tracking system will change the processes used in your company. Let everyone know when and how these changes are happening, and make the resources available to bring them along the learning curve.
Test It
Once the new system is installed and running at your site, you’ll need to run a complete acceptance test. Once your defect tracking tool is deployed, it will be difficult to get a team together to come back later to fix the system. You don’t want to wait two months before finding out that you cannot recover from a backup, or that some entries are disappearing.
Your tests should, at a minimum, represent all your requirements. Your tests should also represent the activities of all of your users. Build use cases for all the principle users of the system—don’t just add a few dummy defects and then close them. Test how easily a programmer can determine what tasks are assigned, and what the priority of those tasks should be. Confirm that managers can quickly gather the data asked of them at staff meetings, as well as the data they require in making decisions. Use cases should include remote and well as local users of your system. And your testing should also address later stages in projects’ lifecycles, after active development is completed, when those projects are being archived and
retired from the system.
Deploy the System
Now that you have the system installed and tested, you need to put it into the hands of your users. During the rollout, training (whether online or in a classroom setting) is important. The greatest tool in the world won’t live up to its value unless effective training programs teach your teams how to use it!
Your staff will claim that they can use the new system and that any techie can learn it in five minutes. Remind them that the new tool is more than just a new GUI to learn. New fields will be tracking different information. Details not even captured in the previous system will now be required fields. You’ve implemented a new defect tracking system, after all, in order to solve the specific problems of the old way of tracking defects—so expect a variety of differences.
Selling and marketing the new system is an important part of the deliverable. Many users of the old system won’t see the reason to change; remember that some of your users may not have considered the old problems as important. Anytime you want others to adopt new technology—even in our field—you should expect some resistance. Be patient; you will need to continue to support new projects as they start using the improved defect tracking system.
Monitor the System
Keep the defect tracking system itself in the database as something users can log defects against. Encourage them to do so when they encounter problems—and be responsive when defects against the system are entered. This helps you learn what problems your users are having with the new system, and what features they would like to have added or removed.
Establishing a good system is only half the work; you’ll need to organize a team to continue supporting the defect tracking system after its deployment. Like any other system, your defect tracking methods will be much more effective if you continually improve them.
Conclusion
Unexpected things happen. Throughout the process of implementing your system, assume that unforeseen changes are inevitable. Even more critical than those surprises along the path is your reaction to them. A core vision will help to sustain your team through those setbacks. Keep in mind that you’re delivering something that will really help your organization be successful, and don’t compromise that vision.
At the same time, realize that there is really no perfect solution, and that all systems will have defects. To believe otherwise would mire you in a cycle of second-guessing your decisions, and—as the old Chinese saying goes—”the best is often an enemy of the good.”
Imperfect at birth, your systems will, with time, acquire even more defects…because requirements are always changing. During the development of your defect tracking system everyone involved will have compromised on their positions, and many design decisions will be reversed (sometimes many times). When the system is delivered, it will be by no one’s approximation anywhere near perfect. Understanding what is right and wrong about the system is important, and the mechanism for continually improving the system should be visible and accessible.
Accepting the idea that what’s being delivered has flaws—and will always have flaws—allows you to get past those imperfections and decide what is really necessary to reach a successful solution. As James Bach has written in Good Enough Quality: Beyond the Buzzwords, you will ship with known problems. Get version 1.0 out and in the hands of your customers, and then worry about what enhancements need to be made.
The work of producing a new and improved defect tracking system for your organization is a commitment to process improvement that may often be tested. To succeed, you must stay focused, be ready to negotiate and compromise, and remain flexible.
