Automation or Not, It's All About the Data

[article]
Summary:

While anyone who has automated her testing knows you can't create repeatable automated tests from unstable data, it did not dawn on this week's columnist—self-proclaimed automation lobbyist Linda Hayes—that this issue cripples manual testing as well. Read on to share her epiphany.

As an automation lobbyist, I constantly whine about test data—or the lack thereof. It's basically impossible to develop repeatable automated tests without a known, stable data state. For companies that are transitioning from manual to automated testing, realizing this is like stepping into an ice cold shower: it wakes you up in an unpleasant sort of way.

Don't get me wrong, I know it's a huge problem. You can't just go around archiving and refreshing monster databases, and even if you could, there are related files and interfaces between applications that make it even harder. It didn't dawn on me until recently that the real problem has nothing to do with automation at all.

Here's the scenario: I was working with a company who was evaluating test automation. As part of the assessment, one of their QA managers walked me through a test case manually. The test was to issue a loan against a 401(k) plan. First she had to find a plan that permitted loans, as well as a participant within that plan who had a sufficient cash balance for the loan, had not taken out a loan within the past year, and did not have an outstanding loan from a previous year.

This took her about half an hour. Once she found the right plan and participant, it took about ten minutes to issue the loan and confirm it was accepted. Granted, she was explaining things to me as she went along, so without my involvement the whole process would have been faster—but the ratio between locating the data and executing the test case would have been the same.

Next it was time to automate the test case, but as soon as we started she pointed out that we could not use the same participant because it now had a loan outstanding and no longer qualified. So, the whole process had to be repeated.

At this point I concluded that automation was impossible, because we would have to essentially write an artificial intelligence system that knew everything she did in order to find valid accounts. Their environment was not stable enough to either reproduce the same data or even let us add our own data, since it was shared by others and updated constantly.

After discussing the implications with her and her manager, we agreed that automation was not an option unless the data environment was brought under control. This would require a substantial investment in terms of hardware, software, and time. I encouraged management to make a business case by pointing out all the benefits that automation would bring. They agreed to run it up the flagpole but made no promises.

It wasn't until some time later, when I was reflecting on this issue for another account, that it suddenly struck me: automation has nothing to do with it!

Tags: 

About the author

Linda Hayes's picture Linda Hayes

Linda G. Hayes is a founder of Worksoft, Inc., developer of next-generation test automation solutions. Linda is a frequent industry speaker and award-winning author on software quality. She has been named as one of Fortune magazine's People to Watch and one of the Top 40 Under 40 by Dallas Business Journal. She is a regular columnist and contributor to StickyMinds.com and Better Software magazine, as well as a columnist for Computerworld and Datamation, author of the Automated Testing Handbook and co-editor Dare To Be Excellent with Alka Jarvis on best practices in the software industry. You can contact Linda at lhayes@worksoft.com.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!

Upcoming Events

Apr 29
Apr 29
May 04
Jun 01