Noel: The abstract for your upcoming STAREAST session "Implementing a Crowd Sourced Test Effort" states that "achieving quality in the current day scenario calls for newer and innovative test techniques." What is it about today's times that require such innovation?
Rajini: Software development processes and methodologies are evolving lately to accommodate changing user needs and requirements. These are core drivers for success in the market place. These core drivers are necessitating several radical changes in how software is developed including:
a. Faster time to market
b. A collective ownership of quality
c. The need to focus on product domain knowledge, understand competing products, creatively emulate end user scenarios and mimic end user environments in test labs
Traditional testing techniques involving a dedicated test team will continue to be used to enhance product quality amidst these changing scenarios. The question is, will they alone suffice to ship a quality product within these constraints that teams operate within? This is why newer and innovative techniques such as crowd sourced testing are becoming an additional need in the current day.
Noel: What specific qualities about crowd sourced testing made you a believer in it, and then a proponent for it to be used on a much wider scale than just your own projects?
Rajini: Crowd sourced testing is a very versatile technique. It has a very simple definition of bringing in the community at large to test for your product, yet is very powerful and scalable spanning across companies (whether product or services, small or large), technologies, domains. It helps bring in diversity into the testing effort, enhance team productivity at very low costs, and strengthen test coverage especially across multiple geographies and devices which might otherwise not be possible to mobilize within an in-house test effort.
I’ve practically seen examples where it has been leveraged in large ISVs such as Microsoft as well as mid-sized services companies such as ours (QA InfoTech). This versatility combined with amazing return on investment, once the effort has been well planned, are what excite me about crowd sourced testing.
Noel: When you're defining crowd sourced testing for groups or companies that you speak with, do you see any variations in the definition, or more specifically, misconceptions that people may have about what crowd sourced testing really is?
Rajini: Yes, in fact one of the core things I address in my sessions / discussions on this topic is to help people understand the varied manifestations of crowd sourced testing.
Often there is a misconception that crowd sourced testing is all about engaging / working with a company whose business model is to pool in a crowd of testers, who get paid for valid bugs they report. Well, this is definitely one form of crowd sourced testing, but there are several other more important and creative manifestations to look at, to leverage the crowd’s wisdom including:
a. Sourcing relevant people from within one’s company although they might not be directly working on the said project
b. Building a pool of end users gradually over time to provide feedback across various phases of product development (be it public beta users, dog-food programs, private betas covering MVPs – Most Valuable Players etc.)
c. Partnering or working with organizations and universities who have domain knowledge and subject matter expertise or are capable of representing the end users in testing your product
The important thing to keep in mind in all these definitions is that money is not what always motivates the crowd. There are other factors such as brand loyalty, transparency and a sense of inclusion in your product before it is released, community involvement etc. that might really excite the crowd in testing your product. Understanding what motivates your crowd and what form of implementation will benefit your project are critical factors in building a successful crowd sourced test effort.