The Role of Artificial Intelligence in Testing: An Interview with Jason Arbon


In this interview, Appdiff’s Jason Arbon explains what the rise of artificial intelligence means for the world of testing. He covers how manual testers can work with AI, the role of automation, and the type of companies that testers can now start.

Josiah Renaudin: Welcome back to another TechWell interview. I’m joined by Jason Arbon, the CEO of Appdiff and a speaker at this year’s STARWEST. Jason, thanks for joining us. First, could you tell us a bit about where you worked at before you started Appdiff?

Jason Arbon: Hi, Josiah, nice to chat with you again. After college, I started my career at Microsoft doing testing and automation for products like Windows and Bing. Later while I was at Google, I worked on test automation for the Chrome browser and ran a team doing personalized web search. Finally, at Applause (formerly known as uTest), I directed product and engineering, focusing on mobile and software infrastructure.

Appdiff is the world’s first AI-powered mobile app testing solution, and we are building smart little bots that have started testing over ten thousand apps today, and soon every app on the planet.

Josiah Renaudin: Something you have a lot of experience with is artificial intelligence. To kick things off, how does AI work, and what’s its current role in testing?

Jason Arbon: Artificial intelligence (AI) is a bit of a mystery and can be intimidating at first, but part of that is because AI is such a broad term. In our context, we’re referring to the ability for a machine to understand an environment, perform “intelligent” actions, and learn how to improve itself automatically.

One of the first AI problems people face is how to find patterns in data, and this has led to lots of classification algorithms, such as neural networks and support vector machines. If you’ve collected lots of examples of how a computer should behave given some inputs, you can “train” bots on this data by showing it the input and output pairs over and over again. After training, the bots are able to do the same task—even on inputs it has never seen before. It is like teaching a child by example.

In Appdiff’s case, we’ve used AI to build and train software bots, which knowhow to tap, type, and swipe through an app—just like a real user. My first industry exposure to AI and neural networks was while working at Bing, and later at Google. Bing was largely powered by AI back then, and my team had to ensure the quality measurements for search engine results were correct. I’ve been straddling the intersection between AI and testing my whole career.

Testing is a ripe field for applying AI because testing is fundamentally about inputs and expected outputs—the same things needed to train bots. Testing combines lots of human and machine-generated data. Folks in testing often don't have much exposure to AI, but that will change quickly, just like everyone else in the world is waking up to the power of AI.

Josiah Renaudin: We’ve heard that “testing is dead” from a few prominent people in the industry, and I feel like a lot of that has to do with automation taking center stage. As AI evolves, will it ever replace manual testing, or will there always be a place for real people actually testing the software?

Jason Arbon: Testers quietly ask that question a lot. The real value in human-powered testing is the creativity required to either identify problems that are subjective or discover bugs that some of the smartest people around (software engineers) didn’t think of or weren’t able to predict at the time of implementation.

In my experience, more than 80 percent of testing is repetitive. You’re often just checking that things work the same way they did yesterday. This work is solvable by AI and automation. That other 20 percent of a tester’s time today, the creative, questioning, reasoning part—that is what people should really be doing, and that rarely happens in today’s fast moving and agile app teams.

Working alongside AI, testers in the near future will be able to focus on the most interesting and valued aspects of software testing.

About the author

Upcoming Events

Feb 27
Mar 06
Apr 24
Apr 24