Unwrapping the Box: Strategies to Approach Mobile Web App Testing

A couple of years ago, I decided to focus on testing mobile applications. Today, I’ve reached a point where the majority of my testing experience is with mobile web apps, and I find myself spending a great deal of time researching approaches, tools, and networking with other mobile testers to get a better grasp of the subject. Given the past of change, I eventually came to the conclusion that this would never stop; that I would need to continually, constantly invest time and effort into keeping up.

Before we jump into testing them, let’s spend a moment or two on the applications themselves. On a high level we have three types of mobile applications—native, web, and hybrid. Each one has its own advantages and disadvantages. Native applications are specifically designed to run on device OS and machine firmware, while web applications pull resources from the Internet each time it runs (mobile compatible websites). Finally, there are hybrid applications that have a native shell and are implemented using web technologies. I personally have worked with all three types and understand that there are multiple factors in each one of them. A tester needs to be aware of this and think about test objectives, test approach, test design, and test execution specific to the type of application being tested.

The current direction in web evolution is “responsive web design.” The simplest way to explain the concept is to say that every website should render “properly” for every form factor of a particular device. Another way to put it is that responsive design attempts to bridge the gap between a desktop browser experience and the web experience on a mobile device by creating websites that work both on a full-scale desktop computer and a mobile device. To do that, the site will have to change dynamically when viewed on a mobile device. If we can get this conversion to happen automatically, then the programmer only needs to create one site, saving development time.

Of course, this automatic translation can have errors; that is where the new testing challenges come in. Users don’t want to see errors and they demand access to Internet on their device. Here are some statistics of just how wide this difference in use is:

• 28 percent of Internet usage comes from a mobile phones. It is projected to take over desktop usage by 2014
• 48 percent of users say that if they arrive on a business site that isn't working well on mobile, they take it as an indication of the business simply not caring
• 62 percent of companies that designed a website specifically for mobile had increased sales
• 90 percent of people use multiple screens to access the web
• About 91 percent of US citizens have their mobile device with them 24/7
• The number of unique screen resolutions have risen from 97 in 2010 to about 232 in 2013
• Only 55 percent of companies are currently conducting any online user experience testing

To sum it up: If your customers are in a position to expect the software to work on “their” devices, then responsive web design, and a test strategy for it, should probably be on your learning list. The world is heading in this direction and it is here to stay at least until the next game changer appears. Some good examples of responsive web sites are the websites of Microsoft, Disney, and the Boston Globe; there are others listed at www.mediaqueri.es

Testing these responsive websites is a challenge as there are different devices, form factors, screen resolutions (over 200 unique screen resolutions), different browsers, and countless permutations. When I wanted to learn to test responsive sites, there was no “book,” so I experimented with different approaches. Here are a few of the things I tried, which resulted in me learning new techniques:

• Networking with mobile testers and consultants both at my company and elsewhere
• Keeping track of new mobile development and testing news and technologies
• Researching different tools to help out in testing
• Evaluated how rapid software testing approaches could help out in mobile testing
• Researching production defects. This is to help to focus testing on types of problems the users are seeing in the application
• Attending mobile focused conferences

Another thing to keep in mind is that because mobile is such a vast domain, it’s important to try out different approaches rather than just sticking to a set of scripted test cases and hoping you have met the so-called “100 percent coverage.” These test cases may be effective in covering some aspects of the application, but they rarely cover the aspect of usability and testing from the end user point of view. I have seen this happen time and again throughout my experience as a mobile tester. Don’t fall into the trap of writing detailed test cases for every scenario. By doing this, you will end up consuming valuable testing time that could have actually been spent on exploring the application. The mantra I follow is to perform “Very high-level test cases complemented with exploratory testing."

Video: Jonathan Kohl’s Keynote at STARWEST 2012 covers a few ways that mobile testing is different than traditional web applications. Watching the presentation and experimenting with Kohl’s ideas is one example of how to develop your mobile testing skills.

The “learn-and-experiment” approach worked well for me while testing my first responsive website. For example, my project implemented totally new pages and functionality that interacted with multiple systems. This being the case, it did not make much sense to waste time in writing detailed test cases as there are only so many interactions we could cover. Instead, we wrote high-level test cases and covered more interaction scenarios by doing exploratory testing.

Finally, never hesitate to challenge anything when working in a mobile web project; always ask questions. No one can predict exactly how a new feature would, could, or will be used in the real world, and this is especially true in mobile and the responsive web design world. There are always challenges, and our job as testers is to challenge ideas and question idealistic thinking.

Speaking of challenging, do me a favor: tell me what I got wrong in the comments section below. I’m sure you readers would love to have more experiences represented than just my own!