Migrating from Test Cases to Real-World Telemetry Measures

Ken Johnston

Ken Johnston sees today’s software ecosystem in the light of Everything as a Service (EaaS). Operating systems like Windows, Android, and Chrome OS all ship regularly like a service. Browsers automatically update every few weeks, and apps are constantly updating through all the app stores. Although getting a test to pass once and signing off has gone by the wayside for software testing, still we run test cases over and over again. Ken shares how Microsoft took millions of test cases—yes, actually millions—and turned the important ones into measures based on real world telemetry. Massive amounts of data coming in from real devices and real users measure product quality and tie it to key customer satisfaction metrics. Join Ken to learn how your organization can shift from internal regression and other testing to dynamic telemetry from the real world and how you can identify just the right number of measures that truly matter.

About the Presenter

A principal manager on the Microsoft core data science team, Ken Johnston and his team focus their research on Windows post sales monetization and device usage within the commercial and education segments. Since joining Microsoft in 1998, Ken has held roles of group program manager for Bing Data Quality and Measurements, group manager for Bing Shopping and Data Operations, test lead and test manager on MSN, and director of test excellence. Ken coauthored How We Test Software at Microsoft and is a contributing author to Experiences of Test Automation: Case Studies of Software Test Automation . Contact Ken on Twitter.

Upcoming Events

Apr 28
Jun 02
Sep 22