Things That Really Matter in Testing—Today and Tomorrow: An Interview with Bj Rollison


In this interview, STARWEST keynote speaker Bj Rollison talks about the things that really matter in testing. He covers the value of testing certifications, the impact of ISO standards on the industry, and the advance of test automation.

Josiah Renaudin: Today I’m joined by Bj Rollison, a testing mentor and keynote speaker at our upcoming STARWEST conference. His presentation is titled “Things That Really Matter in Testing—Today and Tomorrow.” Bj, thank you very much for joining us.

First, could you tell us a bit about your experience in the industry?

Bj Rollison: My career in the computer industry started in 1992 working for a startup in Japan, building custom systems from the motherboard up. In 1994, I landed a position on Microsoft’s Windows 95 International team as a test lead. Over the last twenty years I worked on several products, and also spent about eight years in Microsoft’s Engineering Excellence group teaching and driving practices across the divisions. Since leaving Microsoft last year, I have focused on teaching fundamental testing principles and practices to burgeoning developers to hopefully improve their unit tests.

Josiah Renaudin: You mention in the abstract for your keynote that you’ve often seen people in testing get stuck in trends and topics that make good fodder for debate but do very little to enhance personal careers or the testing profession as a whole. Can you provide examples of these trends?

Bj Rollison: Yes, one example is ISO standards. Sometimes, minimum standards are necessary. But generally they are always minimum standards, and I have never seen ISO standards required in the development of consumer software industry. Some companies may require standards to comply with some governmental requirements, but ISO standards have no impact on the majority of testers in the industry.

Josiah Renaudin: Is agile a modern version of this style of trend, or do you feel it has more value?

Bj Rollison: Agile is a wonderful concept for improving efficiency in a business. But every successful business constantly strives to improve efficiency. Another such concept was poka yoke, or mistake proofing. It was a wonderful concept for an assembly line but didn’t quite catch on in the software industry.

The point is that some concepts for improving efficiency in one business sector may not apply to other types of businesses. The practices necessary to build a successful operating system usually far exceed those of building a game for a smart phone.

Josiah Renaudin: What is your definition of the “traditional” testing role, and how has it evolved over time?

Bj Rollison: The concept of a “traditional” testing role is a complete farce. The traditional testing role was created by very senior and experienced developers such as Boris Beizer, Glenford Myers, and Jerry Weinberg. When I was hired at Microsoft, all testers were expected to have an understanding of operating systems and programming languages. Was that the “traditional" testing role?

In the mid- to late ‘90s, Microsoft and other companies bought into the concept of hiring super-users and customer-like users as testers to find bugs. I suspect many people consider this as the “traditional” testing role—especially since the industry once again seems to be requiring testers with higher technical skills and greater competency who can provide more value throughout the entire development process. The key is to forget “traditional” roles. The industry changes very quickly. If you get mired down in some abstract notion of a “traditional" role, you may soon find that your value to your company or project becomes obsolete and unnecessary.

User Comments

Tanya Watson's picture

BJ, love your take on "traditional" testing.  To me it's a common term thrown around as a means of identifying the role of testers in an organization usually by people with no idea of what that role really is. It's an easy thing to fall back on and not put the work into identifying what the organization really needs.

February 18, 2016 - 2:55pm
Timothy Western's picture

"and I have never seen ISO standards required in the development of consumer software industry" 

Just because you have never seen it doesn't mean it doesn't happen.  In fact we've seen this happen in government contracting where the idea of 'CMMI Rating' became something that people lobby into contracts to reduce competition.

ISO has a pretty good reputation with some of its standards industries around the world use them and strive to include them in some way.  The danger is that when you talk about minimal standards as BJ mentions, you run into the problem that you see on many of these home improvement shows.  Homes which may be 'up to code' but code may not be strong enough, or in some cases, work being done which may look like it passes code until you dig below the facade.  Software development is no different in this respect.

March 23, 2016 - 10:19am
Gregarious Rex's picture


I believe BJ referenced the government as a caveat. <br><br>


Though I do see your point about “minimal” standards. I find with the age of Agile, organizations confuse light standards with no or inadequate standards and this includes what they decide to document. As a traveling consultant with 26 years’ experience, I have seen a rapid depreciation of standards and the rise of poor testing practices masquerading as the "agile' process. You need only move outside of traditional software companies to see this trend first-hand.


March 24, 2016 - 6:09pm
Sanat Sharma's picture

Agree with the point that Automation versus Manual testing is always a grey area for top management in almost all the organizations. When we talk about automation, management always thinks about cost cutting and resource cutting. In my experience of 15+ years in Software indistry, I hardly see a relevant and required automation test suits which can reduce the testing time and improve productivity.

May 20, 2016 - 4:29pm

About the author

Upcoming Events

Oct 01
Oct 15
Oct 24
Nov 05