Managing the Testing Process

Opening the Black Box

Treating testing as a black box at the end of the project schedule invites failure. Testing and test preparation need to be open, managed, and manageable processes that everyone on the project team can understand and work with. Having insight into all testing requirements and desired outcomes is an absolute must for effectively managing a successful testing effort.

Planning for testing on a software project is often challenging for program managers. Test progress is frequently unpredictable, and during software testing painful schedule and feature "surprises" typically occur. Software testing is often viewed as an obstacle-more as a problem and less as a vital step in the process. For this reason, testing is treated as a "black box" and addressed at the end of the schedule. While budget and time may be allocated for it, testing is not really managed in the same way as development. Typically, software development is measured in terms of overall progress in meeting functional and business goals. Software testing needs to be measured in similar terms to understand its true progress and make informed decisions. By considering testing dimensions other than cost and schedule, managers and other team members can better understand and optimize the testing process, in effect opening the black box and managing testing more effectively. In this way they can avoid costly and painful "surprises" late in the project.

Years ago I worked on a new "next-generation" project at a software company. Everything seemed to be going well until the time came to do quality testing. As the software approached the test phase, the schedule looked pretty good. There had been some slips, but the project seemed on target for the scheduled release date. That is, until it actually went into testing.

On this project, testing had been treated as a black box. Software went into the lab, the testers did whatever it is that testers do, and bug reports came out. Since it was a black box, it was difficult for management to understand or justify time, resources, or equipment for testing. Also, nobody outside of the testing group, including project management, really understood what the process was or how to determine whether or not testing was complete. Several weeks were allocated to do a single test run, after which it was anticipated that the software would be ready to release.

Unfortunately, that was not the way the project went. Instead of a single pass through testing, ten different versions of the software required complete testing runs. The schedule was hopelessly overrun, and instead of having a new release to announce at an important show, all the company had were some "closed-door" demos for selected customers.

Testing does not have to happen this way. Looking beyond simple cost and schedule considerations opens up the black box, helping managers make better decisions about testing. When the testing black box is opened, and enough detail is provided to managers, it becomes much easier to understand the tradeoffs and to make informed decisions about resources and schedules needed to test a product. Managers are able to plan for and guide testing from the start, rather than ignoring it and hoping for the best. When the entire project team is kept up to date on testing, understands the approach used, and can observe the progress of testing while it is being done, it is easier for the team to function effectively as problems are found and as new versions go through the testing phases of the project. There are other advantages to opening up testing as well.

Often, there are early warning signs that testing is going to have problems. These show up in the details of the analysis and design phases of the tests themselves. They appear in the form of incomplete or deferred work due to missing information, improperly managed problems recorded against key functionality, and other "small" indicators accumulating over time. If these indicators are spotted far enough ahead of time


StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.