TrainingConferencesAbout UsContact UsAdvertiseSQE.comRSS Feed
StickyMinds.com: brain food for building better software

Log In

Search For:
 Clarify Your Search Criteria

Tips on Using Our Search Feature(s)
 
StickyMinds.com Home
ResourcesTopicsCommunityPowerPassBlogs
Home  >  Detail: Automation Déjà Vu...Again!



A StickyMinds.com Original
Article Picture
Automation Déjà Vu...Again!

By Dion Johnson

Send This Content to a FriendGet a Short Link to This ContentPrint This ContentSee User Comments About This Content

Summary: A decade's worth of test automation history reveals that not much has changed. Experts still dwell on how much to automate and how to estimate different types of ROIs. Test automation's growth is stunted because it's not revered as a discipline different from manual software testing. In this week's column, Dion Johnson urges us to correct the situation so that test automation can develop into a more lucrative opportunity.


Infosys
1997--Cem Kaner's "Improving the Maintainability of Automated Test Suites" white paper:
"When GUI-level regression automation is developed in Release N of the software, most of the benefits are realized during the testing and development of Release N+1."

1999--Bret Pettichord's "Seven Steps to Test Automation Success" white paper:
"We need to run test automation projects just as we do our other software development projects."

1999--Mark Fewster and Dorothy Graham's Software Test Automation book:
"If no thought is given to maintenance when tests are automated, updating an entire automated test suite can cost as much, if not more, than the cost of performing all the tests manually."

2001—Dion Johnson's Designing an Automated Web Test Environment white paper:
"With many, getting an automation tool is like a kid getting a new toy—they jump right in and start playing. And the resulting test suite, much like a child's new toy, just doesn't last."

Time Passes

2008—An unnamed test lead's automation request:
"I know that the application is still changing, but do you think you can start now and automate most of the tests so that they may be used during the acceptance testing in two days?"

Are you kidding me!? More than ten years have passed and we are still at a point where this type of request can be made with a straight face? Why has the industry as a whole not outgrown this? Why do we continue to ask the same questions that we largely asked over a decade ago? For its part, the IT industry and many of us within it have worked to raise the level of automation discourse through the introduction of new techniques, training, and publications. Somehow, this still seems not to have translated to the broader segment of the industry's population.

We are still preoccupied with questions such as:
  • Is record and playback an effective automation approach?
  • Is 100 percent automation possible?
  • How do I calculate return on investment (ROI)?
  • How early can test automation begin?
  • Can test automation replace manual testing?
Don't get me wrong; there's nothing wrong with asking these and other questions, particularly if you are new to IT, software testing, or even test automation. The problem, however, comes when these questions linger, seriously delaying the effective implementation of test automation and, even worse, leading many down the wrong path with regards to test automation. In addition, the over-preoccupation with these questions that have been asked over and over again for more than ten years--despite the fact that some relatively widely accepted answers are available—is one of the major symptoms of test automation's stunted growth. This stunted growth is also evident both in the fact that shelfware remains prevalent and in the missed opportunity to address more pressing concerns. Over the years, we have failed to come up with comprehensive solutions to several important automation issues, such as:
  • Detailed calculations for framework selection
  • Detailed Calculations for Automated Test Development and Maintenance Times
  • Making Risk-based (Quality-based) ROI Calculations More Acceptable
  • Moving to a Fourth Generation Automation Framework
  • Devising a Good Answer for an Acceptable Percentage of Tests that are Automated
Detailed Calculations for Framework Selection
Below is a formula that I often use to help define the level of complexity an automated test framework should have:

 

AF = AN + VN + BN + TN + CN + EN + Ti + P + R

  • AF = Automation Framework Definition
  • AN = Number of applications expected to be tested by your organization
  • VN = Number of versions/releases that each application is expected to have
  • BN = Number of builds/nature of application changes that each application build is expected to have
  • TN = Number of tests that you're expecting to automate for each application
  • CN = Number of configurations that an application may have to test
  • EN = Number of environments in which tests will be executed
  • Ti = Time period over which the tests will need to be maintained
  • P = Organizational process maturity
  • R = Resource technical level
The formula is not meant for literally plugging in numbers to get an answer for AF, but rather simply to illustrate the relationship of the framework choice to the automation scope—a relationship that may be used as guidance for selecting an automation framework. However, the past ten years could've been used to develop consensus within the industry about how this type of formula could be used literally to plug in values for an actual answer to the type of framework that might work best for the identified scope.

Detailed Calculations for Automated Test Development and Maintenance Times
How long should one expect to develop and maintain automated tests? There is a factor that is widely accepted among industry leaders that it will take three to ten times as long to automate a test as it does to execute it once manually. That means if it takes you one minute to execute a test manually once, it will probably take somewhere between three to ten minutes to automate that test. This is still a fairly broad range because there is no industry standard regarding the time it should take to maintain an automated test suite given a certain scope. We need to focus on further defining this information.

Making Risk-based (Quality-based) ROI Calculations More Acceptable
The three basic approaches for measuring ROI are:
  • Simple ROI—Quantifying automation benefits in terms of the monetary savings automated test execution provides over manual test execution of the same set of tests
  • Efficiency ROI—Quantifying automation benefits in terms of the time savings automated test execution provides over manual test execution of the same set of tests
  • Quality ROI—Quantifying automation benefits in terms of the potential monetary savings automated test execution provides via increased test coverage and reduced risk of application failures
Each measure has its pros and cons, but there is one measure that has been largely neglected in practice—Quality ROI. Quality ROI provides a broader view of the benefits of test automation that can get lost if you only focus on the Simple and Efficiency ROI measures. Neglecting the broader view may hurt an automation effort that is already riddled with unrealistic expectations about an immediate, high-monetary ROI. The past decade has seen lost opportunity in effectively moving the industry to a more balanced approach for measuring ROI.

Moving to a Fourth Generation Automation Framework
Automation frameworks may be discussed in terms of three generations:
  • Generation 1 (Linear)—Framework that is composed of automated tests where all components that are executed by a given tests are mostly contained within that automated test
  • Generation 2 (Data-driven, Functional Decomposition)—Framework that introduces more modularization, reuse of common components among several scripts, and greater separation of data from automation code
  • Generation 3 (Keyword, Model Based)—Framework that adds a new level of abstraction that separates code logic from test logic, allowing automated tests to be built and maintained in a less technical way
Each generation has its pros and cons and should be evaluated for use in terms of the automation scope of the organization. There is an opening, however, for a new generation to be introduced. I've dabbled with the concept of building detailed manual test procedures that could be interpreted by an engine and therefore double as automated tests. In addition, there have been some tools that have sought to offer this capability through business components that may be used to build manual test procedures that may also double as automated tests. So while there has been some movement in this direction, it could and should be more pronounced.

Devising a Good Answer for an Acceptable Percentage of Tests that are Automated
The question has been posed for years, "What percentage of manual tests should be automated?" Most people that have been involved in test automation for a number of years can provide some guidance for estimating an acceptable percentage, but the reason this is so difficult to answer is because every project builds manual tests at a different level of detail and also deals with applications of varying complexity. Therefore, 75 percent automation may be feasible for one organization, while only 15 percent is feasible for another. We could, as an industry, evaluate these different conditions, and devise a standardized approach for calculating an acceptable percentage.

The lack of growth in test automation largely is due to the fact that test automation has not been treated as a separate discipline from manual software testing. There hasn't been a separate body of knowledge or a comprehensive resource that focuses on automation processes and procedures, as opposed to just automation tools. We must correct this, so that we can move the discipline of test automation forward and stop the automation déjà vu that we've been stuck in.

For more information, visit Automated Testing Institute.

About the Author
As a senior test consultant and managing partner for DiJohn IC, Inc. and advisor to the Automated Testing Institute, Dion Johnson provides IT consulting services that focus on the overall system development lifecycle, with particular focus on the quality assurance, quality control, requirements analysis and automated testing. He has presented at numerous SQE conferences and contributed to StickyMinds.com and Better Software magazine. Email Dion at dionjohnson@dijohn-ic.com or dionjohnson@automatedtestinginstitute.com.

Back to Top 

StickyMinds.com Weekly Column From 11/3/2008 

Member Comments
Add Your CommentExpand Comments
 
Comment:    
by Gilles Girard 11/18/2008

Not sure what you are driving at but the reality of today is:
Application automation: normally follows developemnt, with the state of development these days (Agile and Xtreme programming) automation remain very basic and Agile/Xtreme automation are requested. You cannot automate programs that are dynamic/changing all the times. You can only create very short scripts ... I hate to admit it but 'record/playback' has proven extremely valuable in these situations (short, easy to change for each builds)
The other value of automation is certainly creating/maintain the test environments (os, browsers, test data, services...Read On

Author's Response: 11/18/2008    
I totally agree with you that Record & Playback has its benefits. I express this in a column that I did called, "Record & Playback, You Have My Apologies". It is particularly useful in agile environments (in which I’ve worked) for quickly getting scripts developed. What I am suggesting, however, is that there needs to be greater understanding of the various automation techniques and frameworks, and what level of verification and maintenance power those techniques and frameworks get you. Surface level automation will yield surface level verification (and in some cases no true verification at all, but rather automated tests that merely exercise the system) and low to no maintainability. Many people don't understand this, however, and think that quick and dirty automation is going to yield astronomical returns, which is usually not the case. As more is requested of test automation, more of an investment is required. Today, however, much of this investment is wasted on concepts that we've been wrestling with for more than ten years. This article suggests that if we work to mature the test automation discipline, and invest less time on old concepts, we can maximize our automation returns no matter what development methodology our projects use.

Thanks again for adding additional depth to the discussion. Stay tuned for increased discussion of this topic at www.automatedtestinginstitute.com.

 
 
Comment:    
by Kawal Sachdev 11/10/2008

I agree with the factors given by you for calculating Automation framework.

Companies should realize that there are some tests which at the end are going to be cheaper for them.i find automated testing non-resistable for performance testing. I think more than the tool, it also depends upon the tester how the scripting can be done to perform automated testing.


Author's Response: 11/10/2008    
Thanks, Kawal!

 
 
Comment:    
by Sanat Sharma 11/9/2008

Well said Dion, about Test Automation and realities.

Some facts that ZAT (Zero Aware Testing) management, (for more, please check - http://xtremeedge.blogspot.com/2008/04/principles-of-testing-and-zat.html) doesn't know about automation are
1. It needs some bandwidth in the initial stage.
2. It is expensive.
3. It is an addition, not a replacement to manual testing.
4. It raises the hopes of higher management.
5. It often frustrates and disappoints the testing team.
6. It delivers a high quality product but can create as many problems as it solves.
7. It should be a full time...Read On

Author's Response: 11/10/2008    
Thanks for the feedback, Sanat!

 
 
Comment:    
by Parvinder Ghotra 11/6/2008

Dion, thanks for another Great article on Test Automation.
Here are few of the reasons why i think Test Automation has not experienced much growth yet:
- It's a very underrated field, management often fails to realize, or in a lot of cases doesn't know, the skill set required
- People continue to compare Test Automation to manual testing, which leads to an inappropriate set of expectations.
- Shortage of people who are good testers and happen to be technical as well. It is pretty rare to find people who fit in that category. Too often, you see people with one or the other, but not both.

Also, i think the...Read On

Author's Response: 11/6/2008    
Good comments, Parvinder! Thanks!

 
 
Comment:    
by Eric Deslauriers 11/5/2008

Dion,

I'd like to suggest that your generations definitions are not quite accurate.

I have yet to see a good MBT solution, and the ones out there are homegrown. It's a great concept, but implementation is still not cost effective for many teams (for instance, we haven't been able to justify it yet, though have some pieces which are "smart" enough to determine if the right outcome was reached).

A GOOD implementation of your Gen 2 is highly effective for a team made up entirely of engineers than MBT or Keyword driven. It's easier to use, faster to deliver results, and easier to maintain. Compared to a...Read On

Author's Response: 11/5/2008    
Congratulations, Eric! You managed to fit an entire article into the comments sections ;-)

I see your point about how you chose to break out the automation generations. I find, however, that there really is no single way in which everyone breaks out these generations, so neither lacks 'accuracy'. I believe that automation professional, Ed Kit in an article dating back to 1999 entitled "The Third Generation--Integrated Test Design and Automation", uses definitions close to the ones that I'm using. Dorothy Graham, in the book that I reference at the beginning of the article, however, talks in terms of 5 generations. Whether spread out across three or five generations, however, the basic definitions remain fairly constant, and that's the first item of important. The next item of importance, however, is where do we go from here? What will be the next generation, and how do we improve upon the existing ones? It seems like you have some excellent thoughts on the topic, and I hope to see you participate in some of the future discussions at the Automated Testing Institute online resource (www.automatedtestinginsitute.com)

Thanks!

 
 
Comment:    
by Dorothy Graham 11/5/2008

Great article, Dion - you make many really good points! It is very frustrating that the message doesn't seem to be getting through much - I have also seen this.

I would like to respond to one of your points - about what percentage of tests should be automated. I agree with you, but there are problems with trying to use this as as an objective for automated tests. It assumes that all of your current manual tests are good candidates for automation - this may not be true. It also assumes that only manual tests can be automated, but there may be tests that are impossible or impractical to do manually that would be really good to...Read On

Author's Response: 11/5/2008    
Thanks for the excellent point, Dorothy!
I 100% agree with you that we must carefully guard against incorrect assumptions being made with respect to test automation, particularly with automated test percentages. The Automated Testing Institute which I’m an advisor for is publishing a Body of Knowledge document that discusses some of these topics in detail (if you’re interested in learning more about this, please fill out the contact form at www.automatedtestinginstitute.com). I do, however, think that if done carefully, it is possible to set a goal for the effective calculation of an acceptable automation percentage, given a certain set of organizational parameters. I believe that the key in doing this, however, is in taking into account all the things you have mentioned, and more. One thing that I’ve done in the past is redefine what “automation percentage” actually means. While it is sometimes important to talk about “automation percentage” in terms of the number of existing manual tests that are automated, it is also important to talk about 100% automation in terms of ‘automatable tests’. This means we identify the subset of tests that are automatable, and use that number as the baseline. Therefore, 100% automation may in actuality be less than 100% of the tests in the test bed. Anyone using an industry standard for calculating automation percentage would therefore have to do a similar analysis of which tests fit into the ‘automatable tests’ category. The discussion of automated tests that don’t have a manual counterpart, I’ll save for another day; maybe another article…
Thanks again for helping to broaden the discussion!


 
 
Comment:    
by Jim Hazen 11/4/2008

"The lack of growth in test automation largely is due to the fact that test automation has not been treated as a separate discipline from manual software testing." - This and the situation where automation is not being properly promoted by the tool manufacturers. The Record/Playback instant gratification snake oil is what is really Deja vu. This has been going on for 20 years (that I know of and have had to deal with), and the question for us is how do we stop this. I'll give my answer another time.

Some people want sciptless tools and no programming, just being cynical here that will never happen. We will always have some...Read On

Author's Response: 11/4/2008    
I liked the Robin Williams quote, Jim!

You make some valid points. I would like to qualify one of your statements about scriptless automation, if I may. You initially state that scriptless automation is not going to happen, and then you indicate that this is because scripting is necessary for making automated tests reusable and robust. I think the first statement has to be joined with the second one to say, "if reusability and robustness are important quality attributes for an organization's automation framework, then scripting will probably be necessary". I do, however, believe that there are certain situations where the automation scope will place less of a need on reusability and robustness, and therefore makes scripting less necessary. So there are some situations in which automation without scripting may be feasible and useful. I do agree with you, however, that over-preoccupation with this desire for scriptless automation, has been detrimental to the automated testing discipline for years, because the automation scope and goals of many organizations really call for more advanced automation skills. We, therefore, definitely need to make sure organizations and potential automation professionals are technically adept, and understand basic scripting concepts and techniques.

 
 
Comment:    
by Linda Hayes 11/3/2008

Great column, Dion. It is frustrating that we still seem to be grappling with basic questions this far along. In my opinion, a fourth generation solution is one that requires no code be written at all. Period. Until we reach this point automation will struggle, because companies are not prepared to write code to test code. It costs too much and takes too long and requires a different set of skills other than testing.

Author's Response: 11/4/2008    
Thanks for the feedback, Linda!

I agree with you that a fourth generation solution that greatly reduces the need for scripting will make test automation more acceptable to many if not most organizations. In many situations, however, regardless of how good a tool's GUI is, I believe that there is no substitute for scripting, because it adds a level of power that helps to greatly simplify tedious tasks. Not to mention, I've seen a strange dichotomy in the desires of organizations with respect to automation. Either they want to be able to automate with absolutely no scripting, or they want pure scripting, with no GUI and no framework (Linear scripts). In the second case, completely removing scripting will make the automator feel as though their abilities have been hampered. Having said that, I totally agree that a fourth generation solution should more greatly remove the need for continuous scripting of basic automated actions, actions that most novice automators will be engaged in. But to ensure that the needs of all automators are taken into account, I believe it would be good to provide automators with the ability to enhance the fourth generation framework with scripting if they so choose.

 
 
Comment:    
by Nitianand Ramjattun 11/3/2008

This is a great article which touches several aspects of test automation. I think that a lack of growth in test automation is also due to the fact that
(1.) Commercial automation tools are too costly while many companies still hesitate on open-source alternatives.
(2.) Many companies do automation only in 'spare time', possibly due to commercial pressure. Leaders need to be convinced on how effective test automation can actually reduce the risks associated with commercial pressure! and thus test automation deserves being a full-time activilty & discipline on its own.
(3.) The manual test design and documentation process...Read On

Author's Response: 11/4/2008    
Great feedback, Nitianand!

I think you add some good points to the discussion of why automation growth has been stunted. Also, I'm glad you are looking forward to the Automated Testing Institute site. I look forward to you joining some of the automation discussions there.

 
Back to Top



 
Ads By Google
What's This?
 
 

Home   |   Resources   |   Topics   |   Community   |   PowerPass



© 2010 StickyMinds.com. All rights reserved.
StickyMinds.com is a division of Software Quality Engineering.
Privacy Policy    Terms & Conditions    Link to StickyMinds.com    Feedback
MicroFocus




STAREAST 2010 


Better Software Conference