execution time of automation suites

Vivek Kumar's picture
Vivek Kumar asked on December 18, 2013 - 7:43am | Replies (3).

How to reduce execution time of automation?

3 Answers

Timothy Western's picture

There are a number of ways to tackle the 'execution time' issue.   It is highly contextual, and also requires an understanding of the type of testing the particular type of automation is doing.

 

I've known teams that disable tests that take too long, and only runs them, manually, sparingly, when they have a reason.  I've also seen teams that instead look at what they are testing, considering what is really valuable and prioritizing those versus a whole suite run.

I hope most people don't end up in a situation where, for whatever reason, they need to stop the automation entirely.  I have known of teams that have done that though, either because of lack of knowledge of the testing, lack of understanding of the testing, or because they were failing more than they were passing, and the needed knowledge was not in house anymore.  For some they may choose to do that along with reprioritizing, or rearranging their test Strategy, which may require writing new tests to replace the old.

 

Beyond that, you can look at ways to leverage hardware concurrency, parallelizing test execution.  This is not a simple programming exercise though, and will not solve all test execution problems either.  Another possibility could be to reduce the sets of test data being used, so fewer test cases are actually run.  Some may use a pairwise tool, like the one produced by Hexawise to reduce the combinatoric matrix significantly.

 

I think questions about test execution time, is a chance to engage stakeholders about what we are testing, why we are testing it, and then discussing what is important now and moving forward. Ideally, this should be going on from day one.

Jeff Downs's picture

Spread out your automation execution across multiple virtual machines so scripts can run in parallel.

Build smarter waits into your code.  Rather than just waiting 60 seconds for an object that may only take 5 to display, use a loop to check every second up to 60 times.  If the object appears, move on.  If not, wait a second and try again.

With Ruby/Watir you can set the execution speed.  You may be able to with other tools also.  Be sure execution speed is set to fast.

Be sure to only run tests that need to be run.  Many times tests are run just because they are automated.  Is there any value in running the script?

Review your startup and teardown framework.  Do you need to run that code for every test case or just at the beginning and end of an automation suite?

Build in proper error handling so that scripts will continue to run around the clock unattended.  A test case failure shouldn't cause your entire suite to stop running.

 

 

Krishnan G's picture
Krishnan G replied on February 16, 2016 - 2:40am.

The way I did was to spwan multiple threads to execute parts of automation suite in parallel, for reporting I used thread safe logging mechanisms like Log4J along with SQLlite database. At the end of all the parallel runs the results were queried from SQLLite database and published in speadsheet.

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.