Maintaining test cases is a tough task when the product under test is changing constantly. This challenge was met by our team and this article explains our experience.
Changes to Product features can happen many times during a product development lifecycle. These may be driven by change of customer requirements, design changes or some times as late as customer acceptance tests. Many times, making necessary changes reported by customers can be crucial. In such scenarios test cases drawn by Test Engineers can become obsolete rendering the whole effort in test planning a fruitless exercise
Planning for Test Case Maintenance is critical as:
- Test cases may become redundant due to behavior change.
- Expected Outcome may change for some test cases.
- Additional test cases need to be added because of altered conditions.
The following is an experience in such a challenge and how it was handled.
Beginning
This was a 3-month project cycle. At the time high level requirements and design were published, our QA team started listing test cases along with all other test plans. Our Team spent a week of effort to list all possible test cases. These test cases were reviewed by development team and approved.
Testing Phase
As development started delivering product our team started executing these test cases. Status of test cases was updated on daily basis and status reports are prepared.
After 15 days of testing the test status was like the following:
Number of test cases: 580
No. of test cases covered: 120
No. of Defects Reported: 10
As the product testing continued, in next couple of weeks development added major features to existing product as they felt these features are necessary. This called for revision of test cases in addition to continuing existing test cases.
As some defects reported also uncovered invalid design and incorrect behavior, the product behavior was gradually changing.
Our team kept reviewing each and every change to code from change notices filed by development electronically. Our team also started making changes to test cases list to reflect the changes to product functionality and behavior.
This dynamic test case maintenance of test cases continued along test case execution for rest of the testing cycle.
End Result
At the end of the 3-month testing, the following is a summary of weekly status of test cases are reported over the testing cycle:
This rigorous discipline to update test cases and updating status of test case and monitoring the results has helped the team meet the challenge of test case maintenance. The following are a few observations from the experience and lessons learned from this exercise. This was also coupled with risk monitoring of test cases as discussed in Article: Approach to Implementing Risk Based Testing.
Observations:
- Number of test cases doubled during the testing phase
- Almost 10% of test cases became invalid during testing either due to changes in product or due to improper understanding of feature and test conditions.
- Test Progress was rapid in beginning and towards the end of the project the % change in progress became relatively small.
- Even during final few days testing cycle there were addition of cases and deletion of some cases.
Summary:
The following are the important lessons our team learned
- It is important to keep the test case list updated to reflect changes of product features otherwise for next phase of testing there will be a tendency to discard current test case list and start afresh.
- Making the Test Case a 'Work Sheet' is very important. On daily basis this test case list should be updated. To enable each member in the team to update test cases the Excel Worksheet was kept in a shared drive and workbook was shared to allow concurrent editing.
- Keeping track of Test Cases Metrics complements Defect Reports Metrics and gives a better view of Product quality. When you read that there are 30 defects pending and only 50 test cases are pending with 1200 test cases successful, you have better knowledge of product quality.
- Maintaining the test case list updated works as proof of testing. In future when a defect is reported, it will be possible to validate whether this case has been covered during testing or not.
- This list of updated test cases list can be used by anyone who will do the testing for next version of product. Same team need not work on next release as test case list matches 'current' behavior of application.
