Weighing the Costs and Benefits of Mobile Test Automation


Today’s market is teeming with attractive options for automating your mobile testing—each with a price tag. You may perceive mobile automation solutions as too expensive, but before you dismiss test automation as a viable solution for your business, let’s break down the benefits, options, and costs of automating your mobile QA process.

Today’s market is teeming with attractive options for mobile testing solutions. However, taking advantage of these solutions in your mobile test strategy requires multiple decisions to ensure your apps delight your customers and perform as expected.

You need to address whether you will use real devices or emulators, a cloud-based or on-premise test lab, and open source or commercial tools and platforms, or a combination of these selections. You’ll also have choices to make concerning coverage for operating systems, browsers, networks, and carriers, and if you want to work with full-time employees, contractors, or crowdsourcing services.

Each of these options comes with a price tag, and the total cost of your testing strategy also will include other factors that are less obvious. You may perceive mobile automation solutions as too expensive, but before you dismiss test automation as a viable solution for your business, let’s break down the benefits, options, and costs of automating your mobile QA process.

The Costs of Manual Testing

First, let’s look at manual testing, which often isn’t as inexpensive as it first seems.

Manual testing is the way most teams get started: you gather several business users, engineers, and anyone else who understands what the app is supposed to do, and you ask them to perform certain tasks on your app. This may be the lowest-cost option (at least to start with), but it certainly isn’t free. Expenses include devices and salaries for employees to run, manage, and organize testing tasks and results.

There are also intangible costs related to human factors. For instance, documentation, which can be time-intensive to write, takes up man-hours. And the limited hours in a workweek cause longer testing cycles and delayed time to market than if you had automated scripts running twenty-four hours a day.

Then there’s the cost of escaped defects. Considering people, not machines, run these tests, it’s easier for mistakes to happen, especially when performing repetitive tasks. And if you do have people doing repetitive work, you potentially risk losing good employees, so you need to factor in the cost of employee turnover.

A final major cost is having suboptimal test coverage and possible additional escaped defects due to the fact that you can only run a limited number of tests with a limited number of permutations. This is big in the context of mobile apps, considering you need to validate multiple app version updates, on different devices, screen sizes, and OS versions, emulating different network types and connection speeds on multiple carrier platforms.

The Costs of Automated Testing

Of course, mobile test automation has its costs, too. When it comes to commercial testing tools, depending on the vendor and the package, your team could expect to spend between ten thousand and more than a hundred thousand dollars a year for enterprise licenses and maintenance contracts. For that investment, you can get premium features, frequent enhancements, integrations and APIs, and comprehensive support.

But simply looking at the price tags for automation tools doesn’t provide the full picture of the total investment required.

You also need to consider the cost of running the tests. Mobile testing best practices recommend testing an app on ten or more different devices, and at several hundred dollars per mobile gadget, costs can quickly add up. An alternative to testing on actual devices is to use emulators to simulate different screen sizes and resolutions, operating systems, and network conditions. Emulators can still run up to about two hundred fifty dollars a month, and they have their own pros and cons compared to testing on actual devices.

User Comments

Pratik Patel's picture

Very Informative Article Michael, related to the importance & benefits of Mobile Test Automation.

As we see the drastic growth in development of various mobile apps for Android & iOS, effective testing of the same is very necessary. Mobile test automation provides various test solutions, which help you to verify and validate the overall functioning of the mobile apps. Today, Automated software testing is considered as one of the most trending term in the field of software testing, as it is considered as one of the efficient as well as effective software methodology, making the overall software development/testing life cycle better.

September 9, 2016 - 2:40am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.