Weighing the Costs and Benefits of Mobile Test Automation


Today’s market is teeming with attractive options for automating your mobile testing—each with a price tag. You may perceive mobile automation solutions as too expensive, but before you dismiss test automation as a viable solution for your business, let’s break down the benefits, options, and costs of automating your mobile QA process.

Today’s market is teeming with attractive options for mobile testing solutions. However, taking advantage of these solutions in your mobile test strategy requires multiple decisions to ensure your apps delight your customers and perform as expected.

You need to address whether you will use real devices or emulators, a cloud-based or on-premise test lab, and open source or commercial tools and platforms, or a combination of these selections. You’ll also have choices to make concerning coverage for operating systems, browsers, networks, and carriers, and if you want to work with full-time employees, contractors, or crowdsourcing services.

Each of these options comes with a price tag, and the total cost of your testing strategy also will include other factors that are less obvious. You may perceive mobile automation solutions as too expensive, but before you dismiss test automation as a viable solution for your business, let’s break down the benefits, options, and costs of automating your mobile QA process.

The Costs of Manual Testing

First, let’s look at manual testing, which often isn’t as inexpensive as it first seems.

Manual testing is the way most teams get started: you gather several business users, engineers, and anyone else who understands what the app is supposed to do, and you ask them to perform certain tasks on your app. This may be the lowest-cost option (at least to start with), but it certainly isn’t free. Expenses include devices and salaries for employees to run, manage, and organize testing tasks and results.

There are also intangible costs related to human factors. For instance, documentation, which can be time-intensive to write, takes up man-hours. And the limited hours in a workweek cause longer testing cycles and delayed time to market than if you had automated scripts running twenty-four hours a day.

Then there’s the cost of escaped defects. Considering people, not machines, run these tests, it’s easier for mistakes to happen, especially when performing repetitive tasks. And if you do have people doing repetitive work, you potentially risk losing good employees, so you need to factor in the cost of employee turnover.

A final major cost is having suboptimal test coverage and possible additional escaped defects due to the fact that you can only run a limited number of tests with a limited number of permutations. This is big in the context of mobile apps, considering you need to validate multiple app version updates, on different devices, screen sizes, and OS versions, emulating different network types and connection speeds on multiple carrier platforms.

The Costs of Automated Testing

Of course, mobile test automation has its costs, too. When it comes to commercial testing tools, depending on the vendor and the package, your team could expect to spend between ten thousand and more than a hundred thousand dollars a year for enterprise licenses and maintenance contracts. For that investment, you can get premium features, frequent enhancements, integrations and APIs, and comprehensive support.

But simply looking at the price tags for automation tools doesn’t provide the full picture of the total investment required.

You also need to consider the cost of running the tests. Mobile testing best practices recommend testing an app on ten or more different devices, and at several hundred dollars per mobile gadget, costs can quickly add up. An alternative to testing on actual devices is to use emulators to simulate different screen sizes and resolutions, operating systems, and network conditions. Emulators can still run up to about two hundred fifty dollars a month, and they have their own pros and cons compared to testing on actual devices.

You may need to change your team or environment to take advantage of automation, too. If you don’t want to do in-house testing, you can set up a cloud-based mobile testing lab. It helps the team run functional tests anytime, anywhere, and test on multiple devices under real user conditions. Access to a full-service cloud-based testing lab is estimated at five hundred dollars a month for a team testing on four different devices. And most open source and commercially available tools out there require knowledgeable people with programming skills to be able to effectively create, run, and maintain test scripts. So you also may need to factor in hiring other mobile application testers.

The Benefits of Automated Testing

Once you get past the initial setup, you will start to see some benefits from automation.

Automated tests run quickly and efficiently. This is especially useful for regression testing, when you have a new version or release of the app and need to quickly check that updates didn’t break the app’s functionality. Manual testing requires similar effort, whether you are releasing a new app or pushing out a quick update, but automated regression testing is fast; you can launch the regression suite from your CI tool and rest assured that nothing is broken.

Data-driving your test automation iterations can potentially uncover more defects in the same amount of time as a number of manual testers, freeing up your people to do more fulfilling tasks. Automated tests also can validate more processes on more platforms, ensuring your app will work on different devices, OS versions, and network speeds.

And automated tests are reusable. You can record a test once and replay it multiple times against new builds, new versions, or different devices. A smart tool will recognize if something has changed on the app screen and will guide you to adjust your script or run the same test against the new layout without requiring modification.

Cost-Saving Strategies and Ideas

If you’re convinced the benefits of automation could help your mobile testing but you’re still worried about the costs being prohibitive, there are options. Building an app is like building a house—costs can vary depending on what you want to build and how you want to build it. The strategies below can help you save money without compromising quality.

Also known as in-the-wild testing, crowdsourcing is a popular way to test your app under real-life user conditions. If you don’t have your own audience of potential users, you can send your app to companies that offer crowd-testing services and they will launch it to their network of testers, coordinate the testing process, and report results.

If the cost of automation tools is the problem, open source tools such as Appium, Calabash, and Selenium are free—but as with all complex digital solutions, you will need to budget additional funds to use them well. Because they are script-based, these tools require that you have skilled people to develop and maintain your testing scripts, so in addition to devices and test lab setup fees, you will need to ensure you have skilled automation engineers on hand. Otherwise, you might spend more time debugging scripts than actually running them.

For teams that want to do test automation but don’t want to learn automated scripting, there are cost-effective alternatives to script-based testing using a new generation of scriptless digital automation tools. You can use your own devices and existing resources to record, enhance, and maintain reusable test scripts. Tests are created as screenshots that provide a comprehensive view of all the steps and actions taken in a test case and are then stored in the cloud. No servers, extra resources, automation specialists, or labs are necessary.

As always, the cost of not testing your mobile apps is far greater than any of the above options. The downstream cost of liability, potential damage to your brand, customer churn, and a low app store rating can cost your company big time. It pays to invest in quality, and there are plenty of available options to help you get on your way.

User Comments

1 comment
Pratik Patel's picture

Very Informative Article Michael, related to the importance & benefits of Mobile Test Automation.

As we see the drastic growth in development of various mobile apps for Android & iOS, effective testing of the same is very necessary. Mobile test automation provides various test solutions, which help you to verify and validate the overall functioning of the mobile apps. Today, Automated software testing is considered as one of the most trending term in the field of software testing, as it is considered as one of the efficient as well as effective software methodology, making the overall software development/testing life cycle better.

September 9, 2016 - 2:40am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.