Crowdtesting: A Personal Account

My company was selected to manage crowdtesting for an advertisement campaign for a very high-profile brand. The client needed assistance in testing an interactive, dynamic advertisement that was to run in forty countries during the 2014 FIFA World Cup. Consisting of twelve interactive variations, the advertisement was also regionally optimized, differing with product offering, language, and localized particulars.

The complexity of the campaign played a large factor in the road to completion of the project. The campaign’s success was dependent on the end-user experience and how the ads were served. Working with a global brand meant there was no room for error, and the international nature of the campaign increased the complexity.

One of the first challenges was handling the time restraints. Within the month we had for testing, we had to cover ramp-up, testing commencement, and testing completion. Because I functioned as the senior test manager for the project, the responsibility for finding resources, managing the project, and discovering all possible defects fell to me.

One of the biggest issues I was concerned about was finding the correct testers in each region with the right skills and the right equipment. Furthermore, as I only speak a limited number of languages, I had to ensure that I could communicate sufficiently with all selected participants. My company has resources in many countries that we utilized, particularly where there was access to the necessary hardware, along with proxy servers. We also used an industry-leading crowdtesting platform, which was bountiful in providing the necessary components. Within four days, I was able to find two relevant testers for each of the forty targeted regions across the world.

To accomplish full coverage, I established a suite of around fifty test cases and recommended that the test team supplement that suite with exploratory testing. Throughout the next couple of weeks, we performed a variety of testing, including functional, localization, and compatibility testing.

I am happy to say that by the end of a month of testing, we found no more defects and none were reported by the customer or end-users. However, it wasn’t easy to get there. Having testers dispersed across the globe meant that effective communication was a challenge, but I worked hard to ensure that everyone on the team knew what needed to be done by a certain date and was provided the appropriate support as needed.

Motivation is a key aspect to crowdtesting. When leading a test team in the same office with a great, long-lasting relationship, it’s easy to enforce the team spirit and ensure that everyone works hard to do the best job possible. It’s a bit more difficult with crowdtesting, where people are working remotely, and I am basically just hoping that they are working effectively! In fact, we had to replace one of the testers about two weeks in because she was never available when I reached out and I wasn’t receiving any feedback from her. One of the ways we motivated crowdtesters was through a rankings system—testers who performed better were ranked higher, making them more likely to be chosen for new projects, increasing their earnings potential.

Another issue of concern was focusing on delivering a comprehensive and quality service. Ensuring that the testers were focusing on both the critical and noncritical tests, as well as not raising duplicate bug defects, was a notable achievement for the project. Eventually, we decided to perform the testing in sprints, where each team in each region was assigned to different aspects of the advertisement, then the bugs were sent back to the client for resolution. After resolution, I would switch each team’s focus so that a different person would be seeing the new section, without the previous defects (hopefully).

In the end, I felt like this project was a huge success. Most of the testers were really excited to perform the testing because the advertisement was so cool. The dynamic aspect of the advertisement was something none of us had really seen before, and the excitement about this innovative technology drove us all to perform the job to the best of our abilities.

Aside from delivering an ad campaign fit for a stellar brand, I gained from managing a global team of crowdtesters in an effective way. My advice to anyone embarking on a project of this sort is to plan ahead, be patient, and always remember what the end goal is.