Software customers and investors beware. You need to look harder at the process of software development. Organizations, in a move to save cash, have scaled back testing beyond the point of acceptable risk. Some companies have eliminated QA resources altogether, pushing testing responsibilities back onto programming staff, which is itself spread too thinly to get it done. As a result, software (and even some hardware) is being released in an untested state.

Times like these may make such cost-cutting look prudent. But what looks frugal on paper now, can in a competitive real world, run to high hidden costs and lost revenues. What's advantageous in the short-term for management may be disadvantageous for customers and investors. Unsuspecting customers are stuck trying to stabilize fragile product, while the high costs for fixing problems in the field are being shifted to investors as a hidden drain on future profits. If you're depending on a software development company either as a customer or an investor, the question of who's doing the testing is vital. Your revenue depends on the answer.

Software is an invisible machine. You can't peel back the screen and look under the hood to see it run. You can't hear the engine labor over creaky algorithms, or see it falter when the modules don't mesh, or smell it when your data crashes and burns. Testing is the only way to find out what you're buying-or selling.

Software is still hand-crafted to solve unique problems and fit different business needs. You can't budget for development without testing, because without testing you can't know if what you've developed even works. And you can't assume that testing is free. It's going to fall into someone's budget-and both parties to a development contract need to know who's responsible for testing what.

A customer these days ought to ask who's testing their software. If the developers are testing it themselves, this is a higher risk. In the first place, testing isn't their job: a career in development doesn't make them experts at this very distinct (and in some ways opposite) skill-set. In the second place, it's hard to be critical of your own work. It's hard to see defects in your neatly woven code: and when your customers tell you the emperor doesn't look dressed, your first impulse is to try to convince them they need Vitamin A.

So, a company that claims to be delivering tested software should show that it has dedicated resources to do the testing. Customers and investors should expect to see an in-house testing organization that's not pruned too small in proportion to the developers. If a company has, say, one test engineer and a dozen developers, you are looking at high-risk, superficial coverage.

If you find your software project testing is outsourced, that's not necessarily bad in itself: any self-contained project can be outsourced successfully, with effective planning and good management. But a software development organization that outsources testing should have a written system test plan, and be able to produce it, so you can make your own judgment about the management. The outsourcing firm should have more detailed technical test plans, and if the work is custom, you might want to have your own technical staff look them over. You should be confident that the development organization is exercising appropriate control over its test contractor, as you are exercising appropriate control over them. Customers need to look at the whole situation, and decide whether a distant level of control lies within your comfort zone of risk. And remember that communication between development and test can become thorny when the functions aren't co-located. The Internet can facilitate this to some extent. But it's still difficult to show a developer a bug, and talk about it, when everyone is miles apart from each other.

As a software buyer you will want to do some level of acceptance testing no matter what. To make sure you're getting value for your money, your technical people and your users need to be involved in the acceptance. I've known customers who actually allowed the software development firm to write their acceptance tests for them to run-which means that serious bugs and usability problems didn't surface until after deployment. Even a good internal test organization can't completely substitute for customers doing real work. If you can't get access to evaluate or monitor the test plans, you can at least volunteer to be a beta test site. If you find out there isn't a beta test-perhaps because the company no longer has a QA group to support one-this is yet another red flag in this buyer-beware market.

As for the investors, I'd like to remind them of a very old metric. For every $1 you spend fixing a defect in the design phase, you'll spend $10 fixing it in the test phase. And if you skip the test phase, you'll spend $100 fixing it after release. A company that's short of cash doesn't want to find itself with too many defects in the field: those multiples of 100 can add up, while customer trust diminishes. Short-term savings quickly translate to crushing long-term expenses. You wouldn't borrow $1 today, and sign an agreement to pay back $100 in six months, would you?

QA is a necessary and economical part of the software development process. The testing function still costs less than the programming function, and QA engineers already know how to do the testing that developers will never fully cover. Cost-conscious investors, like their cost-conscious customers, won't want to bet the farm on over-pruned QA. Be sure you get a complete answer to your question: Who is doing the testing?