Software Test Automation Increases Test Team Performance

Software testing is a vital yet costly part of the software development lifecycle, and automated testing of software, if implemented properly, can be an important part of software testing. The software development lifecycle is defined as "the period of time that begins when a software product is conceived and ends when the software is no longer available for use. The life cycle typically includes a concept phase, requirements phase, design phase, implementation phase, test phase, installation and checkout phase, operation and maintenance phase, and sometimes, retirement phase. These phases may overlap or be performed iteratively, depending on the software development approach used," (IEEE, 1990). Software testing is an objective view into the code and requirements and verifies that the quality of the software meets end users goals before releasing the product.

Software testing ensures the software meets the requirements as specified by the end user of the software. As pressure mounts to release software to the marketplace quicker, many companies rely more and more on automated software testing to test more effectively. "Manual techniques are performed by people, and automated techniques by the computer," (Perry, 1995). A human must write the automated test case to make sure that the tool will execute the test case properly. Even though the computer executes the test case in an automated test, the tester actually has to do the thinking to teach the computer how to test the software.

Automation testing is beneficial for the time-consuming task of regression testing. "Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirements," (IEEE, 1990). Many times, one small code change to a software system can introduce software bugs to seemingly unrelated parts of the system. Regression testing accounts for a big part of the software testing lifecycle, and test teams are continually trying to decrease costs associated with it. One strategy is the purchase and implementation of automation-testing tools. These tools, at a high level, are like a tape recorder. The tester hits the record button, executes the test cases in a web browser, and then hits the stop button. The test can then be replayed at any time to execute the test case. The use of automation tools can increase the number of test cases run during a test cycle increasing testing and code coverage, which ultimately improves the quality of the software.

Automation can also simulate different users on the system. Let's say a tester logged into the application as "John Doe", but there are fifty usernames that need to be tested due to unique username and access privileges. Some system users may have administrator rights, for example, with the ability to see different functions in the application than other users. Automation scripts can be created to simulate these fifty different users and their associated roles. The tester can record the first iteration with “John Doe" and then add the other users that need to be tested into the automation tool. The automation tool can then be taught to expect the different results the web application would return. In this scenario, a test that normally would have taken hours to execute manually, can now be completed by the tool while the tester is not at their desk. This allows testers to actually get work done when they are not at their desk, which increases the overall value of testing to the organization. Also, testers can focus more on testing new products, which generate new revenue for the company, instead of costly maintenance on an old product. Software automation testing tools enhance testing scope, efficiency, and consistency for any software testing team. By properly implementing an automation test methodology, increases in test team performances can be realized.

There are several key reasons to automate some, but not all, software testing. Such things as usability testing, which is testing the software to validate that the software meets the user's work style, needs to be done by humans because the computer can't judge the flow, look, and feel of a web application (Kit, 1999).

The first reason to automate is consistency. Automation scripts execute the same way, regardless of who starts the previously written scripts. This cannot be said for manual testing. Human perceptions, judgments, and objectivity come into play when humans perform manual testing. What one tester perceives as a software bug, another may not (Kelly).

Second, automation increases worker efficiency. Testers can multi-task more effectively when an automation tool has been implemented. A tester can launch the pre recorded test scripts while he or she is in a meeting, and if an error occurs, that tester can be paged with a notice of failure. A tester might also start scripts, then do test planning on another project, making them more productive.

Third, automation increases reusability of test scripts. Automation scripts can be used over and over again on different operating systems and browser combinations, increasing testing coverage. It becomes time consuming and frustrating to execute fifty test cases on three different operating system and browser combinations. An automation tool will not get frustrated with this task. The tool will avoid errors that humans make when they get tired or bored with a task. When the tool finishes with one combination, it goes to the next combination (Kelly).

Bret Pettichord, a consultant, trainer, and writer specializing in software testing and test automation, states there are three keys to getting test automation kicked off the proper way: commitment, teamwork, and an early start (Pettichord, 2000).

If the test team does not support the test automation effort and each team member is not committed to the effort, the effort is likely to fail. Many times, the test team perceives the automation effort as an extra step in the testing effort that reaps no rewards. Testing teams usually have fallback plans in place to revert to the manual process and typically the automation effort doesn't get the attention that is deserved. On the other hand, when dedicated team members see the effort may be failing, they work hard to get the job done, showing their commitment to the testing effort. Commitment is a very important aspect of having a successful automation effort, and without it, many efforts fail (Pettichord, 2000).

Testers and developers need to work as a team when implementing a test automation effort. Testers need developer support, and many times when developers are excluded from the team, problems arise. Developers can help identify some shortcuts that will make writing the automation scripts more effective. Developers who run the automation effort also need testers to be part of the automation team. Developers usually write fancy automation test scripts, but those scripts usually do not meet the requirements of the testing effort. Developers attempt to find defects in their own code, but neglect the code of others. Testers can bring an independent view to the team, and testers are experts in determining if all the product requirements are being fulfilled (Pettichord, 2000).

Getting an early start to the automation effort can help alleviate major problems in the long run. For instance, if code changes need to be made to assist in the automation effort, it is much easier to make the changes earlier in the software development lifecycle. As the software matures, it becomes more difficult to change the code without major software redesign. The earlier a test team can begin writing automation scripts, the better off the project will be (Pettichord, 2000).

Standards need to be set and followed when implementing an automation testing effort. These standards should allow team members to write their automation test cases in a consistent format. This will ensure testers can view each others automated test cases and understand what another has written. If fellow software testers need to assist with automation test case design, he or she can do so without deciphering what the previous tester was writing. There will be some obvious differences in how testers write their test cases, but naming conventions can be agreed upon and comments should be written in any code to assist in future maintenance.

It is documented that 80% of all test automation efforts fail (Boehmer & Patterson). Many times, testing teams underestimate the effort that is really involved in leading an automation effort. Test automation can be very time consuming and expensive. "It takes three to ten times as long to create and run an automated test as it takes to run the test once by hand," (Kaner 1997). A test automation effort is not an easy task to successfully implement. A test team needs to have a solid plan in place and can expect to put in a lot of work on this effort (Boehmer & Patterson). Without having realistic expectations, the test automation effort will likely fail.

Because there is a large investment in implementing an automation effort, there should be a good return on investment. To do so, automation test cases should be used for regression testing and re-used in subsequent releases of the product. The goal of testing software is to find bugs, but this is not the reason to implement an automation effort. The reason to implement an automation effort is to save time and money and make the testing execution piece of the software testing lifecycle easier and less mundane (Boehmer & Patterson).

Testquest.com, an international software testing company specializing in software test automation, and whose clients include AOL, FedEx, Nokia, and Motorola, among others, lists several factors to consider when looking at software test automation costs and return on investment:

1. Automated testing is faster. Automated tests takes less elapsed time than manual tests simply because automation executes test steps faster than humans do. Further, automated tests do not suffer from the cumulative effect of the minor distractions and delays that are inevitable with human testers: they don't take breaks, don't need to eat, don't take calls from their family, and so forth. Our experience shows that automation can easily execute a given test scenario from 25% to 50% faster than the same tests conducted by humans.
2. Automation can execute tests 24 hours a day, 7 days a week. With manual tests, the tester is involved from beginning to end, and nothing happens unless the tester is physically pushing the buttons. With an automated test, the tester only needs to start the script and evaluate the results. This means that organizations that currently use a single 8-hour shift for their test cycles will immediately realize a 67% reduction in elapsed time by going to 24-hour/day testing. Factor in weekends and holidays, and the benefits get even larger. To get a feel for the impact of this improvement, consider that a human tester works about 2,000 hours per year out of a potential of approximately 8,760 hours. If the organization can fully exploit 24x7 testing, then the compression of the test cycle will be over 75%.

Testquest provides some excellent examples of the gains that can be realized from the automation effort once it has been implemented and testing scripts have been written and are stable. Even though there is a lot of work that needs to be done to implement the automation effort and create the scripts, the long-term benefit is definitely worth the effort.

Test teams must carefully decide what software to automate. Boehmer & Patterson have set up the following criteria to assist in deciding what to automate:

1. Automate regression tests: Since the return on investment in automation comes from script re-use, you should first target your regression tests. Automate any tests that will have to be run with every build (e.g. Build acceptance tests, installation tests, or other high priority tests). Only automate tests that will be repeated. One-time tests are not worth automating.
2. Automate tests for stable applications: Before automating the testing of an application, look at the application itself. There is no point in beginning automation on an application that is likely to change in the future. If the application changes, the automated tests must also change, and re-work is never good (or fun). Therefore, only automate tests for applications that are stable.
3. Automate non-time dependent tests: Do not automate tests with complex timing issues. Many times able bodied automators feel the need to code their way out of complex situations. While this may be fun, it is also a big waste of time. If a test is too hard to automate run the test manually. Automation is not always the answer, and100% automated testing should not be the goal. You want to increase productivity, not waste a week on the automation of a test that takes 5 minutes to run manually, so leave the overly complex tests for manual testing.
4. Automate repetitive tests: If a test is extremely repetitive and boring, please, please, please automate it. The sanity you save alone will be worth the investment. Test zombies never prosper.
5. Automate tests that have been written: You must first have detailed test cases that are repeatable before you start automating. Always write test cases before automating them. This ensures that tests are written in terms of application functionality, independent of the automation effort. If tests are written by automating the automation becomes the focus rather than the testing.
6. Limit your scope: Don't try to automate everything. Achieve small successes then increase your scope as you make progress. Most software development teams are all constantly in and out of schedule crunches. Make the most of the time you have, and grow your automation effort when you can.

Boehmer & Patterson's six-step approach is an excellent foundation for any test team that is thinking about embracing an automation effort. Test teams need to have rules in place that defines what to automate. Test teams must understand what needs to be automated and what does not. Teams should not waste time trying to automate software applications that should not be automated at all. This can cause frustration in any test team and is a waste of testing resources.

Many managers believe that automation will take care of all their testing needs, but this is not true. Automation changes the complexity of the testing team and testing life cycle. Test approaches, testing tasks, and staffing must be considered when implementing an automation effort.

There are significant costs and benefits related to starting an automation effort (Hoffman 1999). The startup costs of the automation effort must take into account the organization's need to become familiar with the test automation effort, which may include some training, the price of the actual product, and the time it will take to write the automated test scripts. Once training has been performed and the product has been purchased, savings to the testing effort accumulate quickly (Testquest). The up-front price of the product is usually a fixed cost, meaning that cost of the product is static, or doesn’t go up or down for use. Most of the human costs, such as writing the test cases and training are variable costs, meaning that the cost of labor goes up or down with the time allocated to the automation effort. Once the test team becomes more familiar with the product purchased, it will take less time for them to write test cases, therefore bringing down the overall cost of the effort.

One way to measure the effectiveness of a test automation effort is metrics. "Software metrics provide a quantitative basis for the development and validation of models of the software development process. Metrics can be used to improve software productivity and quality," (Mills, 1998). Metrics provides the test team with a status of the testing at any given point in time. Management finds quantifiable pass/fail ratios very helpful in determining the state of the software. Important decisions can be made from this information, such as putting the application in a production environment or waiting until there are fewer bugs in the system to go live with the system. “Existing metrics techniques such as code coverage can be used to estimate or compute test effectiveness before and after automation. Automated tests can be incredibly effective, giving more coverage and new visibility into the software under test. It also provides us with opportunities for testing in ways impractical or impossible for manual testing, yet conventional metrics may not show any improvements. Automated tests can generate millions of events and sequences limited only by the machine power and time available for running the tests. These tests can find defects in code already 100% covered.

Employing random numbers allows sampling of events and sequences, and also allows tests to do new things every time they are run. Automated probes can look inside the product being tested at such things as intermediate results, memory contents, and internal program states to determine if the product is behaving as expected," (Hoffman 1999).

Software testing is a very important part of the software development lifecycle. Poor quality software and software that does not meet end users' goals would be the result if software testing were ignored. One challenge that many testing groups have is delivering quality software to the end users more efficiently. Many testing groups look to an automation approach to solve this problem.

The immediate benefit of an automation approach is to reduce human intervention for regression testing. Automation test scripts should be created to regression test stable software. When code changes are made to the stable software, the automated test cases can be run, and if any bugs were introduced in the code, the automation test cases would find them, thus, saving the test group time. When regression testing needs to be done, a tester will not have to sit at the machine and manually execute the test cases.

Test teams that are considering the implementation of a test automation solution should remember the three keys that Brett Pettichord lists: commitment, teamwork, and getting an early start. Without any of these, the automation effort can easily fail.

Standards must be followed when embracing an automation effort. Standards on writing scripts, naming scripts and where to store scripts need to be followed. This will bring some consistency and efficiency to the automation effort. Testers will not have to search for the location of test scripts or wonder what a script is because of an ambiguous name. The groundwork needs to be set before running into problems with inconsistent practices. This will save the team time in the long run.

Test teams should not underestimate the effort and investment costs associated with implementing an automation effort either. The tool will need to be purchased, people will need to be trained, and immediate results may not be noticed. This may frustrate many testing teams, causing them to abandon the effort.

Realizing the cost and commitment level, and having management support is very important to making an automation effort a success. /p>

An automation test solution cannot take care of all the testing needs of any group, and everyone on the test team should realize this. Many testing teams expect an automation tool to solve many problems for them. While this may be true, an automation tool cannot solve all problems the team faces. Testing teams must realize when, what, why, and how to automate before even purchasing a tool. An automation solution greatly changes the complexity of any testing team, and if the team is not aware of this, the team may be set up for failure. Coming up with an automation strategy, even before writing automation test scripts, is a good idea for any test team.

Increases in test team performance will be realized by properly implementing an automation effort. Test teams must understand there is a significant investment with embracing any automation effort, and gains in productivity may take some time. In the long run, an automation effort will bring testing consistency, reusability of test scripts, and worker efficiencies to the team.

References

Boehmer, B., Patterson, B. Software test automation: Developing an infrastructure designed for success.

Hoffman, D. (1999) Cost benefits for test automation. Paper presented at STAR West 1999. Software Quality Methods, LLC. Saratoga, CA.

Institute of Electrical and Electronics Engineers. (1990) IEEE Standard computer dictionary: A compilation of IEEE standard computer glossaries. New York: (IEEE).

Kaner, C. (1997) Improving the maintainability of automated test suites [1]. Paper presented at Quality week 1997.

Kelly, D. Software test automation and the product life cycle. (Vol. 13 Issue 10)

Kit, E, (1999) Software testing in the real world. Harlow, England: Addison Wesley

Mills, E.E. (1988, December). Software metrics: SEI curriculum module SEI-CM-12-1.1. Pittsburgh: Carnegie Mellon University, Software Engineering Institute.

Perry, W. (1995) Effective methods for software testing. New York: John Wiley and Sons, Inc.

Pettichord, B. (2000) Three keys to test automation.

Testquest, Inc. TestAutomation and return on investment.