One Experience with Ubertesters, a Mobile Test Management Platform

[article]
Summary:
Many companies creating mobile apps struggle to find the time to test on a variety of devices, organize bug reports, and resolve issues efficiently. Andrew White’s organization tried Ubertesters, a platform that provides a team of mobile testers and a set of features for feedback. This is his account of how it affected their test process.

Our company, like many other small software development companies, faced the common "time and resources" issue when it came to manually testing our own mobile applications. As the number of projects increased, the number of bug reports and unstructured documents and files grew as well. We also lacked a standard way to build the applications or to submit bugs. All these issues began to slow down projects.

And then there were the changes. We were using TestFlight, a beta testing tool, for build distribution and crash reporting, but then it was acquired by Apple and stopped support for the Android platform. In search of alternatives, we switched to Ubertesters, a cloud-based platform that gives developers access to a crowdsourced team of mobile app testers with a range of tools and devices.

This is the story of that experiment, along with step-by-step instructions on how to execute a project in this space.

Here’s how Ubertesters works at a high level:

  1. Create your organization and project within the platform.
  2. Integrate Ubertesters’s software development kit with your build.
  3. Upload the build.
  4. Add the devices to test on
  5. Add members to your project. These can be developers, testers, or any other people who can contribute to the project.
  6. Share the build with your testers (internal or external).
  7. The tester gets a push notification that a new build is available for testing and installs it to his or her device.
  8. The tester sends reports with the attached screenshots of bugs found.
  9. The bugs are fixed.

Now, here’s how to actually use this platform.

First, you need to create an organization in the Ubertesters workspace. The organization is a set of projects written by developers for different devices.

Create a project within your organization for each new application. In order to do this, you need to choose what operating system your app was designed under. You also can simply drag and drop .apk or .ipa files to the window.

When working with iOS, the developer must add the unique device identifier, or UDID, of the device to the Provisioning profile in iTunes. If the device is not added to the profile or the UDID is wrong, the tester will not be able to install the build. Forgetting this step was our main issue when working with Ubertesters, but the tech support team was extremely fast with their help, so we had no significant pauses at this point.

In order to start using Ubertesters, download and integrate the necessary SDK. If you are writing a cross-platform application, you will also have Xamarin, Unity, PhoneGap, and Appcelerator among the supported frameworks.

Add devices with the appropriate operating system to your project. After connecting and configuring the devices, invite members into the project: developers, testers, and project managers.

If it is necessary to collect statistics on the specific category of users (such as for our outsourcing testing team), we create a separate "distribution group" for them and form it as we wish.

Once a new build is uploaded, testers who are added to the project get push notifications, download the build through their devices, and start the testing process.

Advantages

One pleasant surprise compared to similar platforms is the set of features for faster feedback from the testers. Testers now submit bugs directly from the device with the provided screenshots and description.

With previous tools, we faced the common stupid issue of testers spending more time making reports than testing. If we were lucky, a bug report came as a text description, and then we had to take the time to specify the exact place a bug appeared.

The screenshot editor that is built in to the Ubertesters SDK allows the tester to take a screenshot, draw a circle or arrow to point out the problem, crop, add a description, and submit the report to the platform directly from the mobile device. In addition to the images, all the necessary information about the device, such as the model and screen resolution, is collected automatically.

You can also export bug reports to other bug trackers, which is very convenient for those who work through Redmine or JIRA, for example.

It is also possible to track sessions of every particular device or participant of the project, in order to check how long the session was active and how long it just hung in the background. Though we do not use this feature, I think it would be an ideal solution for project managers who are trying to control the entire process of development and testing.

The Ubertesters console

In the Wild

After two weeks of successfully using the Ubertesters trial version, the speed and quality of our application testing have increased significantly.

In general, the Ubertesters platform is a decent service that can be compared to TestFlight, with a range of additional features such as support of test cases, build management, and automatically collecting the information about testing devices. It also speeds up the testing process because you do not need to send emails with each new build; testers get all notifications immediately.

I hope my experience will help you make your mobile app testing process more efficient and productive.

What tools and tips do you recommend to save time when testing mobile apps?

User Comments

3 comments
Mike Dichiser's picture

How many testers did you end up using and what was the overall cost to use them?
Would you use ubertesters again?

August 6, 2015 - 2:35pm
Andrew White's picture

Hey Mike, sorry just saw your comment. As far as I concerned we've hired about 50 of their testers in total, paying $3000 - $5000 per month to cover all our testing needs (mainly a need for different mobile devices / OSs). They charge $25 per testing hour. 

We will deffnitely use them again in the future, since it's a kind of smart and cost-efficient solution in many cases. For example, once a new mobile device appears on the market, it always generates a high interest among our clients ('- Have you tested my app on iPhone 6s?! Apple released it yesterday!'). With ubertesters we don't need to buy it just to test on it for a couple of hours - we hire testers who already have this device. The same happens with different locations and mobile providers - we just give a call to our project manager there. So in general, we're satisfied.

October 5, 2015 - 12:06pm
Andrew White's picture

Hey Mike, sorry just saw your comment. As far as I'm concerned we've hired about 50 of their testers in total, paying $3000 - $5000 per month to cover all our testing needs (mainly a need for different mobile devices / OSs). They charge $25 per testing hour. 

We will deffnitely use them again in the future, since it's a kind of smart and cost-efficient solution in many cases. For example, once a new mobile device appears on the market, it always generates a high interest among our clients ('- Have you tested my app on iPhone 6s?! Apple released it yesterday!'). With ubertesters we don't need to buy it just to test on it for a couple of hours - we hire testers who already have this device. The same happens with different locations and mobile providers - we just give a call to our project manager there. So in general, we're satisfied.

October 5, 2015 - 12:08pm

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.