The highest risk that software development organizations face today is allowing defects to creep into the releases made for their high-paying customers. At times, either due to fear of losing the order, or due to overconfidence, organizations allow defects to ship. Time to market and (practically) defect-free products are the two pillars of any successful business. This article reiterates the important points in the defect-free part of that equation. How do you successfully implement system testing to ensure that the doors for future business remain open?
It is well known that the cost of fixing defects detected in the later part of the software development lifecycle (SDLC) is much higher than those fixed in earlier stages. Unfortunately, reviews and unit testing often take low priority due either to improper estimation or to the rush towards completion. Often reviews are cut back or skipped because the product will be tested later. Fixing defects in later stages of SDLC not only increases cost, but also creates side effects that might not show up until after delivery.
This fact creates a tremendous amount of pressure on the system testers, as this is the last opportunity the organization has to prevent defects from getting into the hands of customers. It is necessary for an organization to block defects at this stage. The business consequences are obvious for those who analyze the effects of an escaped defect going out the door to the customer. The following points are intended to help analyze put the defect-prevention case into perspective.
When the Customer Becomes the Tester
Most often the defects that escape from testing end up with the customer. This puts more pressure on your customer as he loses confidence in the products delivered to him. This means the customer ends up testing your products. The cost associated with "customer testing" is very high and can lead to deterioration of the business relationship.
Fixing Defects in the Field Is Expensive
There are three aspects to look at in this case. One is the cost associated with the delay, or downtime, it causes the end user. Another is the cost associated with the logistics of the developers and testers in the field. The third is the cost associated with the side effects of the instant fixes made in the field since only a minimal amount of testing is done.
Recalls Should Not Be an Option
Assume that you are a development company of software that resides in your customer's hardware. A new hardware is released in the market with your software bundled within. A hardware malfunction is detected and your customer is forced to recall all the hardware sold in the market. If the cause of the fault is attributed to the software you supplied, then you are expected to compensate a portion of the loss. In extreme cases, you may have to liquidate a portion of your assets to deliver payment. Think of all the opportunities during the development lifecycle when this could have been prevented!
Effort and Cost in Securing the Order Is High
Getting business from a good customer is a real achievement. The amount of effort and money that has gone into getting that customer can be astronomical. An obvious defect that has escaped through your system testing has the potential to do ruin the project.
Defects Are Credibility Issues
When you are an international business, any flaw in your system exposes you to damaged credibility with the international business community. Customers who placed an order, and the new orders waiting in the pipeline, will come to a standstill, as confidence in your company's reliability declines. Can any one measure the loss? Can you quantify the effort and money required to bring back lost confidence? There is an inevitable connection between loss of credibility and loss of revenue.
Future Revenue Losses Are Not Measurable
Defects undermine trust. Loss of customer trust can never be measured. Who knows the future business the customer had planned for you based on your past reputation? Who knows the orders that were waiting, but will now be cancelled? Who knows the business ahead of you based on the quality of your beta release? Future opportunities evaporate when defects are allowed to slip through the exit point.
The only way to reopen the doors of business is to block the defects at or before the exit point. Here are a few methods your team can adopt to help ensure your organization's future success.
Follow System Testing Practices
System testing is an effective filtering mechanism that should be employed prior to delivery. Since system testing is user-level testing, most of the defects that users would see, will be uncovered here. Functionality of the product and the user interface under normal and abnormal conditions, load conditions, and stress conditions, are tested.
Provide Adequate Training to Testers
Train your testers on the concepts and technology involved in developing the product. They should interact with the customer and also function as end users. Motivate them to look at the product and test it the way a user would. Recognize the testers who discover critical defects such as application or system crashes, failure of abnormal user scenarios, performance degradation, or other usability issues.
Create a Neutral Atmosphere
Encourage testers to report any deviations from the specification that are discovered. Advise them not to worry about development issues and schedule pressures. Create an atmosphere where testers can look at the product independently and report the defects. This unbiased and focused testing is essential to uncover most of the defects that are invisible to normal eyes.
Create Backward and Forward Awareness
System testers are sandwiched between developers who think that their job is done when the product is given over for testing and the delivery folks who are trying to snatch the product from the tester's hands to meet their commitments. It is the responsibility of management to facilitate testers by creating proper awareness regarding the importance of blocking the defects at the exit point.
Involve Testers at the Beginning of the Development Lifecycle
In many organizations system testers are not considered as part of the team, they are added as additional ingredients at the time of testing, or the product is given to the testing team once it is ready for testing. In many cases, the products are tested by those who don't have enough prior knowledge of the product or how it was developed. Since the system testing lifecycle starts after the requirement specification is finalized, with test planning and designing, it is essential for the test team to be involved in the early stages of development lifecycle. Early association gives higher levels of confidence to team members.
Empower and Involve Your Testers
Create awareness among testers that their role is important towards opening up more avenues of business. Give them the freedom to report defects as they arise. Involve them in the process of deciding on releases in case of open defects. Listen to testers to decide whether it is an acceptable behavior or not. Encourage testers to suggest improvements to the product under test, which can be incorporated in future releases. The product is closely observed by testers during the lifecycle, so they are in a position to offer constructive suggestions.
Tighten Your Release Process
There is a great deal of pressure to release the product at the end of system testing. Testers play a vital role in ensuring the closure of reported defects and in analyzing the impact of the open defects. Release has to be recommended based on the type of release being made (commercial, beta, evaluation, etc.) and the impact of the open defects on overall quality, considering the user community.
Embrace the testing staff and make sure they have an avenue of communication with business-decision makers. This will open the doors to greater business success.
The Cost of Software Quality by Rex Black (first of a six-article series on Investing in Testing).