Demystifying Mobile Testing: Critical Thinking Prevails

There are so many testers who think mobile testing is completely different from other types of testing. But there is no need to totally shift their testing paradigm and strategy; mobile testing is not that different after all. This article details the commonalities and differences in testing mobile applications and how lessons learned from testing other applications can be applied to mobile as well.

In my years of performing mobile testing and connecting with colleagues through meet-ups, blogs, articles, and conferences, a common theme has always emerged: Testers are nervous or skeptical about getting into mobile testing.

I have often heard testers say mobile testing entails a totally different thought process from other types of testing, and they believe they have to take extra courses and get certifications to be proficient in it. There are even testers who think they are not qualified to do mobile testing, although they have been testing other applications for more than a decade and have gained a lot of experience during those years.

I remember testers saying the same things years ago about testing responsive websites, where a website should render correctly for every form factor of a particular device. In the end, testers with critical thinking prevailed and were able to test these websites and provide valuable information for the stakeholders to make decisions. The same thing is happening now in the era of mobile.

Technology is constantly changing, and it is understandable to feel out of place. But at the same time, we need to realize that, as testers, we can adapt to any domain or technology. Critical thinking and experience-based skills apply to almost any arena.

Mobile Testing: Not So Different After All

Yes, there are differences in mobile compared to other applications in terms of the way it is implemented, distributed, and consumed by end-users. But in terms of testing, it is not that different after all. The concepts, approaches, and strategies testers use when testing other applications can be applied to mobile as well.

Some of the commonalities in terms of areas to be tested are log files, rendering issues, performance, consistency, storage, memory issues, caching problems, and security vulnerabilities.

There are also overlaps in terms of testing strategies that can be used, including:

  • Blink test: Looking for visual patterns by constantly switching between similar versions of pages or apps to notice minute differences in rendering or visual elements
  • Installation testing: Installing, uninstalling, and reinstalling apps, including upgrading apps from much earlier versions
  • Interrupt testing: Testing how an end-user will use the app by constantly texting, calling, or switching apps on the test phone. The same could be done for web pages
  • Testing with different configurations: Testing apps on different mobile devices and OS versions. Similar to browser testing, which involves testing in different browser versions or testing in desktop applications with different versions of the client’s application
  • Checking for consistency: Checking for app consistency between Android and iOS. Similar to looking for consistency in pages by testing between browsers, between browser and mobile, and between desktop applications
  • Checking user reviews: Reading through user reviews to find out how end-users feel about your app and how they use it. This applies to desktop applications and websites as well
  • Checking for rendering issues: Checking whether web pages display differently based on different mobile browsers and screen sizes

There are many examples of issues I have found following the above strategies in mobile and other applications. Whenever I test an application, I always check for consistency with its previous versions, with its competitors, between OS versions, between browsers, and much more.

Once I was testing a booking application that makes hotel, flight, and car reservations via a mobile app. There were both an Android and an iOS version of this app. I noticed that when a customer searched for hotel rooms for more than four people, they were navigated to the mobile website from the Android version of the mobile app; but on the iOS version of the same app, they were able to search for hotel rooms within the app itself without being redirected to the mobile website. The behavior was inconsistent between their versions of Android and iOS apps, and this was a bad customer experience. Once I pointed this out, we were immediately able to change the flow on the Android app to ensure the apps have consistent behavior.

Another time, I was testing a revamped version of a screen capture desktop application. The new features were really cool, but when I checked for consistency between the new version and older versions and other competitor apps, there was a vast difference in the main functionalities. Imagine this situation: The user opens a word processing application and tries to save a file. Any user would immediately look at the top left corner of the screen to click on the File>Save option. This is a de facto standard for word processors. But say the option is all the way at the bottom right corner of the screen. How would that experience be? The same thing happened with the new version of the client’s screen capture tool; the option to capture a screenshot was at the bottom right corner of the screen instead of the top portion of the screen, which the users were used to seeing. Similarly, there were other major buttons and options on the GUI that were misplaced and scattered all over the application. All in all, the application was quite inconsistent with its previous versions and competitors, which would probably result in a poor user experience.

It is important to know these commonalities to understand that mobile is just one type of application, and the testing techniques and approaches picked up from testing other applications are not a waste. These are rich skills that are used across various domains. In my personal experience, I believe that these skills are learned through practice rather than attending a course or getting certified.

Mobile Testing Differences You Should Know About

Now that I have covered the shared characteristics between mobile and other applications, it is also important to realize the differences. There are specific aspects of mobile that, when included in the testing strategy, could give better coverage. Some of these strategies include:

These include switching between Wi-Fi and 4G; testing for specific types of keypads to display for different types of fields; testing device features such as the gyroscope, accelerometer, and camera based on whether the app has features related to gaming, navigation, scanning, or other related functionalities; and testing how much battery life is consumed by an app.

For example, when the alarm goes off in the morning and I wake up, the first thing I do is check Facebook on my phone. At this time, I am on my home’s Wi-Fi network. I get ready for work and get into my car, and now the connection switches from my home’s Wi-Fi network to 4G. Now, I reach work and enter my office building, and the connection switches from 4G to my office’s Wi-Fi network. Then, I take the elevator to the third floor, and my cell phone does not have Wi-Fi or 4G in the elevator. Finally, I reach my desk and the connection is back to my office’s Wi-Fi network.

How many times did my connection switch between Wi-Fi and 4G? This is a normal scenario that happens in many people’s daily lives. Imagine the user is also using our client’s app on a daily basis. How is the app going to handle the constant switching of networks? You can see that it is really important to use this strategy to check the true behavior of an app. This helps us catch caching issues, unwanted crashes, transactions going through when they shouldn’t, loss of data, and other issues.

What Are You Waiting For?

Being aware of the commonalities and differences between mobile testing and other kinds of testing helped me test different mobile web, native, and hybrid applications. In my experience, exposing testers to this kind of information gives them the knowledge that mobile is just another application that needs to be tested, and that the same strategies and skill set used with other software can be adapted to this domain.

I hope this article will instill confidence in those testers who want to get into mobile testing. They may realize they already have the necessary skills to apply all the testing strategies and approaches.

User Comments

John Ruberto's picture

Nice article.  Do you have any thoughts on performance testing in mobile, what are the similarities and differences to web as you mention?    

With many mobile apps, which are designed to accomplish a specific task, you might be in that app for just a few seconds - which makes performance very important.  What are your thoughts on performance testing? 

November 14, 2016 - 2:53pm
Raj Subrameyer's picture

Hi John,

Thanks for your feedback. I am glad that you found my article interesting and useful. 

Talking about performance testing, I am no way an expert in it but based on what I have done so far here are my thoughts.

When we talk about performance, no matter whether it is a mobile native app or a web app, these things are common performance characteristics we would want to investigate-

  • Response times
  • Network bandwidth
  • Memory usage
  • CPU usage
  • Website/Application/Page load times
  • Server side vs Client side calls
  • Caching issues in terms of trying to load previously accessed resources from the cache instead of making another server call which adds to load times an slowness of the app

Some performance characteristics which are more relevant to mobile native apps and woud be interesting to investigate are

  • Battery life- How much battery does the app drain?
  • Specific focus on usage of Maps and Dynamic content, as they make the app really slow. Need to be really cautious in using them in your mobile apps
  • Memory leaks

Tools I have used for mobile web performance testing include

  • Load UI (Open Source)
  • Apache JMeter (Open Source)
  • Blazemeter
  • Neoload

Tools that I have used for profiling mobile native applicatons include

  • Xcode's - Instruments developer tool. This is part of the IDE and is for iOS Mobile applications
  • Eclipse/Android Studio/IntelliJ - adb developer tool - This is part of the IDE and is for Android Mobile applications

In summary, there are quite a bit of tool options to do performance testing. It all boils down to 2 things-

1. What kind of information you are looking for?

2. What kind of applicaton you are testing?

This will help to find the best tool or approach for your performance testing. Also, people often fail to give importance to performance testing and learn the hard way through bad user reviews. So, it is always good to include performance testing as part of your regular testing process.

Hope this helps, if not let me know. 

November 15, 2016 - 11:30am
John Ruberto's picture

Wow, great list.  Thanks for your reply!



November 15, 2016 - 6:30pm
Pratik Patel's picture

Great insights related to the mobile testing. As we see the drastic growth in the daily development of different mobile apps, which can be related to Android or iOS, mobile app testing through automation is in great demand. Mobile test automation provides various test solutions, which help you to verify and validate the overall functioning of the mobile apps.

March 3, 2017 - 4:18am
Raj Subrameyer's picture

Great point Pratik. Mobile Test Automation is indeed useful to solve many testing problems. In my mind, they are no different than other types of automation where we identify mundane and repetitive tasks which takes a lot of manual effort from testers, and then try to automate them while the testers can use their creative thinking in trying to find and explore other vulnearabilities in different applications instead of these mundane tasks.

The key with automation is Cost vs Value. If we think the value is more than the effort of building them and maintaing them, then that is a good reality check to decide on automation. 

Talking specific to mobile, I still think we have a lot of mobile test automation solutions out there but just like other automation tools they have their own merits and de-merits. Also, when deciding on mobile test automation tools another factor to consider is whether to go with vendor tools or open source tools. 

So yeah a lot of things to consider before doing any kind of automation :-) 

Thanks for your input Pratik. I appreciate it.

March 4, 2017 - 3:05pm
Yana Co's picture

Thanks for this very well thought-out article. While a mobile app is technically just another type of software and plenty of general software tests and techniques can apply to it, you went so much further in revealing the unique factors mobile testing has to take into account. From understanding how a mobile app will behave on the go as a device switches from network to network to evaluating the effect an app has on a smartphone's battery life, it's clear that mobile testing is exceptionally user-focused.

May 24, 2019 - 6:20am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.