Hidden Assumptions: What You Don't Know Can Hurt You


Old software may not always work as well as it seems. The mentality of "If it isn't broke, then don't fix it" could be the culprit. In this column, Linda Hayes offers a few suggestions to help you look at your software with a more critical eye, which might help you realize where your old software is broken or in need of attention.

I have long excoriated capture/playback as an ineffective test automation approach because it lacks structure, reliability, and maintainability. But there is something about it that is more subtle and disturbing. When you get right down to it, all capture/playback can do is tell you if the same steps got the same results, not whether the steps or results were right or why they were repeatable.

Batch testing is also often automated by simply capturing production data, declaring it the baseline, and thereafter comparing against it. As expedient as this is for describing a lot of functionality in shorthand, it unfortunately means that an error memorialized in the baseline can go undetected for a long time and do a lot of damage.

Don't get me wrong, I can understand the attraction, even the need to do this. After years in production, any system documentation becomes long out of date and the original authors are perhaps long gone, so the best source of knowledge about what the software should be doing is what it is doing. After all, if no one is complaining then it must be working. Right?

Not necessarily. HomeSide Lending, a New Jersey mortgage broker acquired by National Australia Bank, was under-pricing mortgages for years due to a "computer error" in which the wrong interest rate was used. The eventual write-offs, after all the dust settled, amounted to $4 billion, and the bank's market value lost over $6.5 billion. The bank was almost bankrupted.

Aside from the shocking price tag, it's amazing how long the error went undetected. Mortgage pricing is essential to HomeSide's core business and alleged to be their special competency. I don't know precisely how it happened, but obviously someone, somewhere, and at some time decided the software was correctly pricing mortgages when it wasn't. No one questioned it until the Australian banking authority audited the software.

It stands to reason that the best and only way to truly test software is to first decide what it should do, then prove that it does as expected instead of the other way around. This is hard, especially for older systems, because of the long accumulation of changes and the tangled network of integrations. Still, is any excuse adequate against the possibility of long term, perhaps even fatal damage?

Sarbanes Oxley wasn't around in 2001 when the HomeSide Lending disaster occurred, but if it were, I'm sure the repercussions for executives would have been far more serious than being fired. And while decades ago a "computer error" was easier to cloak in some technical mystery, today technology is commonplace and errors are understood as defects that have a cause.


About the author

Linda Hayes's picture Linda Hayes

Linda G. Hayes is a founder of Worksoft, Inc., developer of next-generation test automation solutions. Linda is a frequent industry speaker and award-winning author on software quality. She has been named as one of Fortune magazine's People to Watch and one of the Top 40 Under 40 by Dallas Business Journal. She is a regular columnist and contributor to StickyMinds.com and Better Software magazine, as well as a columnist for Computerworld and Datamation, author of the Automated Testing Handbook and co-editor Dare To Be Excellent with Alka Jarvis on best practices in the software industry. You can contact Linda at lhayes@worksoft.com.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!