Unwrapping the Box: Strategies to Approach Mobile Web App Testing


Mobile web technology has been continuously changing over the past few years, making “keeping up” challenging. In this article, Raj Subramanian covers the latest trends and changes happening in the mobile web and how testers can prepare for them.

A couple of years ago, I decided to focus on testing mobile applications. Today, I’ve reached a point where the majority of my testing experience is with mobile web apps, and I find myself spending a great deal of time researching approaches, tools, and networking with other mobile testers to get a better grasp of the subject. Given the past of change, I eventually came to the conclusion that this would never stop; that I would need to continually, constantly invest time and effort into keeping up.

Before we jump into testing them, let’s spend a moment or two on the applications themselves. On a high level we have three types of mobile applications—native, web, and hybrid. Each one has its own advantages and disadvantages. Native applications are specifically designed to run on device OS and machine firmware, while web applications pull resources from the Internet each time it runs (mobile compatible websites). Finally, there are hybrid applications that have a native shell and are implemented using web technologies. I personally have worked with all three types and understand that there are multiple factors in each one of them. A tester needs to be aware of this and think about test objectives, test approach, test design, and test execution specific to the type of application being tested.

The current direction in web evolution is “responsive web design.” The simplest way to explain the concept is to say that every website should render “properly” for every form factor of a particular device. Another way to put it is that responsive design attempts to bridge the gap between a desktop browser experience and the web experience on a mobile device by creating websites that work both on a full-scale desktop computer and a mobile device. To do that, the site will have to change dynamically when viewed on a mobile device. If we can get this conversion to happen automatically, then the programmer only needs to create one site, saving development time.

Of course, this automatic translation can have errors; that is where the new testing challenges come in. Users don’t want to see errors and they demand access to Internet on their device. Here are some statistics of just how wide this difference in use is:

  • 28 percent of Internet usage comes from a mobile phones. It is projected to take over desktop usage by 2014
  • 48 percent of users say that if they arrive on a business site that isn't working well on mobile, they take it as an indication of the business simply not caring
  • 62 percent of companies that designed a website specifically for mobile had increased sales
  • 90 percent of people use multiple screens to access the web
  • About 91 percent of US citizens have their mobile device with them 24/7
  • The number of unique screen resolutions have risen from 97 in 2010 to about 232 in 2013
  • Only 55 percent of companies are currently conducting any online user experience testing

To sum it up: If your customers are in a position to expect the software to work on “their” devices, then responsive web design, and a test strategy for it, should probably be on your learning list. The world is heading in this direction and it is here to stay at least until the next game changer appears. Some good examples of responsive web sites are the websites of Microsoft, Disney, and the Boston Globe; there are others listed at www.mediaqueri.es

Testing these responsive websites is a challenge as there are different devices, form factors, screen resolutions (over 200 unique screen resolutions), different browsers, and countless permutations. When I wanted to learn to test responsive sites, there was no “book,” so I experimented with different approaches. Here are a few of the things I tried, which resulted in me learning new techniques:

  • Networking with mobile testers and consultants both at my company and elsewhere
  • Keeping track of new mobile development and testing news and technologies
  • Researching different tools to help out in testing
  • Evaluated how rapid software testing approaches could help out in mobile testing
  • Researching production defects. This is to help to focus testing on types of problems the users are seeing in the application
  • Attending mobile focused conferences

Another thing to keep in mind is that because mobile is such a vast domain, it’s important to try out different approaches rather than just sticking to a set of scripted test cases and hoping you have met the so-called “100 percent coverage.” These test cases may be effective in covering some aspects of the application, but they rarely cover the aspect of usability and testing from the end user point of view. I have seen this happen time and again throughout my experience as a mobile tester. Don’t fall into the trap of writing detailed test cases for every scenario. By doing this, you will end up consuming valuable testing time that could have actually been spent on exploring the application. The mantra I follow is to perform “Very high-level test cases complemented with exploratory testing."

Video: Jonathan Kohl’s Keynote at STARWEST 2012 covers a few ways that mobile testing is different than traditional web applications. Watching the presentation and experimenting with Kohl’s ideas is one example of how to develop your mobile testing skills.

The “learn-and-experiment” approach worked well for me while testing my first responsive website. For example, my project implemented totally new pages and functionality that interacted with multiple systems. This being the case, it did not make much sense to waste time in writing detailed test cases as there are only so many interactions we could cover. Instead, we wrote high-level test cases and covered more interaction scenarios by doing exploratory testing.

Finally, never hesitate to challenge anything when working in a mobile web project; always ask questions. No one can predict exactly how a new feature would, could, or will be used in the real world, and this is especially true in mobile and the responsive web design world. There are always challenges, and our job as testers is to challenge ideas and question idealistic thinking.

Speaking of challenging, do me a favor: tell me what I got wrong in the comments section below. I’m sure you readers would love to have more experiences represented than just my own!

User Comments

Rajini Padmanaban's picture
Rajini Padmanaban

Raj - what is your experience with mobile performance testing? As you talk about responsive web design, one of the key elements that is gaining importance in mobile performance testing is to look at the rendering aspect on mobile devices as opposed to worrying too much on where the load was generated from. Any inputs on this from your testing experience?

October 18, 2013 - 4:09am
Raj Subramanian's picture
Raj Subramanian

Hi Matt,

Apart from those mentioned by you some of my favorites are-

1.Xcode’s in built iPhone simulator to get quick feedback on new features that are implemented

2.Use in built browser tools (Click on F12) in your Chrome and IE browser. Helps to quickly find errors in Javascript, HTML, CSS

3. Switching between Wi-Fi/4G connection. Very simple but very effective in finding hidden bugs

I use other testing approaches too, that helps me to get quick feedback about the system under test


Hi Rajini,

The experience I have in mobile performance testing till now is using Browser tools, Keynote's MITE and Fiddler to monitor page load times, page size, local storage and HTTP calls.

You are right that with "Responsive Web Design" you have to worry about rendering issues, page load times etc.

I think when you talk about Rendering issues, it is more related to user experience of the website rather than mobile performance. When we talk about mobile performance the things which come to my mind based on my testing experience are

1. What are the files which are causing longer page load times? - The usual suspects are JQuery or other javascript libraries which are all loaded in the client side rather than having code on the server side.

2. How does the website pages perform under Wi-Fi/4G connection, other connection providers?

3. How much traffic can the mobile website handle?

4. Are there any memory or storage related issues in terms of servers running out of memory?

5. Why is one page loading faster than the other? What external systems are interacting with the pages

and so on....

Hop this helps :-). If not feel free to e-mail me at [email protected]


October 16, 2013 - 10:11am
Matthew Heusser's picture
Matthew Heusser

Thanks Raj. This reminds me of something I'm working on right now - a list of mobile testing tips that are "hard" - like about:debug in android to catch error messages, or using screen capture (both buttons) in iPhone to get screen problems. What are you some your favorites?

October 14, 2013 - 9:03am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.