Test is a very valuable discipline in the engineering field. For years, there has been a need for test plans to ensure that systematic testing occurs on projects, test designs to ensure the test approach is reflective of the project under development, and test procedures to ensure testing is being performed in a repeatable manner. More recently, the test function must be able to support more and more complex application development. With this comes the need for test automation and more sophisticated test tools that allow for more efficient testing. This implies more test artifacts that must be manage,d which include numerous test scripts, test cases, test data, and test results.

As more test items exist, there is a tendency for them to evolve due to the changes to requirements and code; these, therefore, must be managed effectively. When there are an increasing number of test items, this increases the risk of failing to accurately track all test items, particularly when done manually. The risk of over-writing changes to test scripts due to a lack of control is increased, as is the risk of missing important test results about how software is tested when compared with past or future testing results. As test teams invest more time in effort in the test process, test tools, and test artifacts, there should also be an investment in the control provided on these items. This is where configuration management (CM) can help.

Where CM Can Help a Test Team

CM benefits test by ensuring the code that is migrated to test is properly version controlled and is clearly identified and baselined. There is another way that CM may benefit test, though. This is by establishing configuration management (CM) for test items. Today, this is as important as establishing CM for development. As requirements are changed (added, modified, or deleted) throughout an application lifecycle, this constitutes the creation of new test scripts, the modification of existing test scripts, or removal of test scripts within a baseline. Therefore, effectively managing test items becomes very important since the application and the associated test items continue to change. Also, as more automated test tools like Mercury’s Quick Test Pro, LoadRunner and Rational’s JTest get used, CM would be useful to help recognizes all of the artifacts created. This is particularly true since each of these tools create multiple files in multiple directories for a single test. CM can help mitigate and reduce the risk associated with change by helping a test team baseline and manage their test items and changes to them. It can also improve communications within a test team when everyone knows where to look for test items and changes can be quickly identified. What are some ways CM can help in the test area?

 

·       CM helps to align the correct version of a test script with the correct version of code. Requirements will often change (e.g., get added, modified, or deleted) and this initiates a change in the code. This then may require a change to test scripts that trace back to the requirements they test. This helps ensure quality is systematically improved. Using the incorrect version of a test script will produce test results that can be misleading and lead to unneeded rework.

·       CM helps track the test results so that improvements or regression can be identified. Either a missing or an incorrect copy of the test results can result in incorrect comparisons and lead to additional work. This is particularly true with capture and playback tools that record the events performed during a test session into a file that can be replayed against the same or an updated version of the software, allowing a comparison of identical test results. It is important to identify any difference to indicate if regression has occurred.

·       CM can help track the baseline of test items (test plan, test design, test cases, test scripts, test results, etc.) so that it is clear which items go together. Effectively, these items can be treated as a release of test items for a project. This helps to improve the integrity of the test items as they relate to the requirements, code, and to each other (e.g., other test items) and can also decrease test effort.

·       CM helps control the work on test scripts, which reduces the duplication of effort. Several people may be working on the same test script and copying over each other’s work, which can lead to missing tests. CM ensures that testers are aware of changes as they get version controlled.

·       CM technology provides a common repository for test artifacts promoting a standard location. This also promotes reuse when new test items need to be generated, since previous test items can be more easily found.

Understanding that CM can help solve some of the control related challenges of test is a good start. Once we understand this, though, what can CM or test professionals do next?

Steps to Providing Test with Better Control

Test professionals should understand enough about the benefits of CM to request a CM capability to help their test team. CM professionals should be ready and willing to help test teams build a CM capability. When helping a test team build a CM capability, it should be approached in a similar manner to approaching a development organization. Test groups will need similar CM procedures and a CM technology. Below are tasks that may be considered in building a CM capability that is right for test. Many of these tasks are explained in much more detail in the book, “Software Configuration Management Implementation Roadmap”.

 

·       Perform an SCM analysis of the test environment: Identify the needs of the test team. Determine if there is a test tool in place and learn what test items need to be tracked. In particular, learn which test items need to be associated with other test items. Identify if there are any CM procedures currently being used. This task helps provide an understanding of where the test team is today in relation to CM and where they would like to be.

·       Develop a CM implementation plan: Based on input from an analysis, outline the tasks needed to build the desired level of CM capability for test. This task helps provide an understanding that this is a planned project with expectations of participation and levels of effort.

·       Help select a CM technology: Based on requirements for a CM tool, help the test team to identify the best CM tool. If there is an existing CM infrastructure within the organization, division, or group that can be shared and it fits the Test team’s needs, then consider using this to save time and effort from implementing a CM infrastructure (in fact, it will save considerable time and effort).

·       Define a label convention and branching & merging process/procedure. This helps the testers identify the latest (or baselined set of) test items and provides a way to work on concurrent versions of a test item for parallel development needs.

·       Define a SCM version control process/procedure: This helps the testers know the expected procedure and commands when using the CM technology. For test items, version control is the basic level of control while change control is the advanced level of control.

·       Define a problem management process/procedure:  This helps the testers track defects. This is typically already in place, but should not be forgotten. Also, if there is not already an automated problem management (defect tracking) technology in place, this should be considered.

·       Define a change control process/procedure: This helps the testers understand the change procedure for test items under formal change control (if desired). For test items, version control is the basic level of control while change control is the advanced level of control, but requires more involvement from the test team. This may include performing change control training to ensure everyone understands the process.

·       Establish the CM technology: This includes installing the CM tool, setting up a working infrastructure, and setting up the testers’ workspaces in which they can make changes to test items.

·       Import all test items into the CM technology (repository): This focuses on identifying all test items (some of which should have been identified in the SCM analysis task) and either importing or checking them into the CM technology.

·       Perform CM training: Train all applicable testers in the usage of the CM technology (how to checkout/checkin, how to label related test items, etc.). Conduct a CM system readiness review. Prior to converting to the CM infrastructure (technology and processes), meet with the test team to establish their readiness for using the CM technology and processes.

·       Cut-over to the CM infrastructure: In part, this is a milestone when the test team begins using the CM infrastructure. However, the other part is to be ready to provide intensive CM support for the next 2-4 weeks while the test team gets used to the new CM infrastructure.

Another task where CM may help the test team is participating in the selection of a test automation technology. The CM person can contribute by investigating the integration with the CM technology. This is particularly useful since many automated test technologies produce numerous test scripts, data, and results. It’s good to have the ability to version control them as they get generated and associate related test items.

Summary
Test teams must manage more and more test items every day. CM professionals should be ready to help the test team when they are looking to build a CM capability for their test items. A solid CM process and technology to support a test team will help them manage their changes. This is particularly useful when test scripts must align with the ever-changing requirements. Any change to requirements precipitates a change to the code, which must have a test script or test case to validate the change. If the test script is not controlled, the test script may test too little and leave a gap in testing.  Inversely, it may test too much and incorrectly showing errors. CM can help solve this issue and provide test teams with a solid change infrastructure that reduces the risks associated with a lack of control.

References

1. Chapter 4. “Establish an SCM Infrastructure for an Application” Software Configuration Management Implementation Roadmap, by Mario E. Moreira, 2004, Wiley, Ltd. Publishing

3. “Surviving the Top 10 Challenges of Software Test Automation” by Randall W. Rice, Rice Consulting Services, Inc, published in CrossTalk: The Journal of Defense Software Engineering, May 2002.

3. “Configuration Management – It’s Not Just for Source Code” Ralph Capasso, Rational Software, May 23, 2003.