Testing COTS: When, How, and How Much?

[article]
Summary:

Testing processes and practices are well defined and generally understood for internally developed applications, but what about those that are licensed from third parties? Granted, the vendor has responsibility for testing its own products, but the possibility of the software failing still exists and can be costly, even devastating; blaming others offers little consolation. If you rely on a commercial off-the-shelf (COTS) application, where does your trust in the vendor end?

While there are no hard and fast answers to how much trust you should instill in commercial, off-the-shelf applications, there are some basic guidelines that can help. Start by asking yourself these questions: 

  1. What is the risk that the application has defects?
  2. What is the risk that your use of the application could cause errors?
  3. What is the potential cost of a defect or an error?
  4. If you do test, what type of testing should you do?

Depending on the answers, you may find that your COTS application-testing needs and approach will vary widely.

Application Risk
The risk inherent in the application itself is a function of its scope of functionality, its breadth of users, and its maturity. For example, most would agree that mass-market utilities such as word processors, spreadsheet programs, slide presentation programs, and graphics tools pose little inherent risk. Their scope is narrow, they have millions of users, and most are highly mature.

Then there are more specialized applications, such as accounting systems, that are still somewhat narrow in scope but whose maturity and customer base may vary widely. Consider that there are mass-market accounting systems, niche products that address smaller segments such as non-profit accounting, and even custom solutions for highly differentiated customers. These products may also have wide variations in maturity. A new product for a small market with few customers poses inherent risk.

And then there are massive applications, such as enterprise resource planning systems (ERP), that encompass multiple departments across many industries and support complex, critical business processes. These have a wide scope of functionality, may have many users, and may or may not be mature. Due to their scope and complexity, there is inherent risk in these systems. These risks may be greater for newer products with fewer customers or for customers in a new industry or using a new component.

Usage Risk
The way you use an application can also change the picture. Consider a spreadsheet used by a construction company to manage multimillion-dollar bids. In this case, the software itself may have low inherent risk, but if the template contains complex formulas and is tweaked often, the risk of errors rises.

Many desktop utilities also can be programmatically integrated with other software. For example, an internal stock trading application might integrate a spreadsheet for its graphing capabilities. In this case, the risk of the spreadsheet malfunctioning might be low, but the potential that the integration has errors is higher. The wrong data might be mapped to the formulas, for example.

The usage of enterprise applications is usually controlled by configuration settings and master data. However, the rules that govern a business process can vary significantly from one company to another, and the sheer complexity of these applications with their multiple points of integration raises the risk of error dramatically.

In other cases the user might actually modify an application to accommodate unique requirements. Clearly this type of usage poses the greatest risk, regardless of the stability of the original state of the software.

Tags: 

User Comments

1 comment
Annica Odina's picture
Annica Odina

Important to find out off what shelf the product came... There doesn't appear to be a 'legal' definition of what is actually 'off-the-shelf' and recently I tested a product that the vendor considered 'off-the-shelf' because they were done developing it but it turned out that it was actually not in commercial use! After 2 years of unsuccessful testing the customer (Government Agency) finally gave up and started looking for a truly off-the-shelf product that came with references from existing, commercial users.

August 8, 2012 - 3:16am

About the author

Linda Hayes's picture Linda Hayes

Linda G. Hayes is a founder of Worksoft, Inc., developer of next-generation test automation solutions. Linda is a frequent industry speaker and award-winning author on software quality. She has been named as one of Fortune magazine's People to Watch and one of the Top 40 Under 40 by Dallas Business Journal. She is a regular columnist and contributor to StickyMinds.com and Better Software magazine, as well as a columnist for Computerworld and Datamation, author of the Automated Testing Handbook and co-editor Dare To Be Excellent with Alka Jarvis on best practices in the software industry. You can contact Linda at lhayes@worksoft.com.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!