In traditional testing, with a separate test team, the test manager or test lead was the team leader who made the test plan, organized testing, and supported the testers. But most organizations don’t have these teams anymore. Testers are part of the Scrum team or DevOps team, and testing is also done by others, like developers, users, or the product owners.

Has the test manager role disappeared?

The old role is disappearing for sure. But we see another, different role coming up: test assurance officer and test coach.

In this article we describe two cases where the role of test assurance officer and test coach added value to the project or organization. We don’t see this role in every organization or every project, and the way it’s implemented is different from situation to situation. Let’s see how the role can be useful.

Case 1

Case 1 is a health care organization that rarely does an IT project. The IT organization is relatively small and the experience with IT projects is limited.

The system implemented is a standard application that had to be configured. The processes needed to be defined and implemented before deployment of the system so the system could be implemented by administrators and tested by the users. But although the users had domain-specific health care knowledge, they did not have any testing knowledge or experience. To overcome this obstacle, the organization hired a test consultant in the role of test assurance officer and test coach.

He started with some traditional test management activities like a product risk assessment. He decided how the tests were documented (in this case, mind maps were used for test design), made an overview of functionalities, and, together with the users, they decided who would test what.

During development the users started the preparation—this was the part where most of the coaching took place. The coach taught the users the basic test design techniques relevant for this project and supported the users in applying these techniques. This also led to the first test assurance activities: the test consultant reviewed the test designs, made sure the coverage and quality of the test designs were good, and helped the users to apply test techniques properly.

Some users did not have the time to do the preparation, and some others didn’t feel comfortable with it. In those cases the test consultant did the preparation, mostly together with a user who had the required knowledge of the processes. The test consultant was also responsible for setup and maintenance of the test and acceptance environments.

During execution as test coach, he monitored the progress and made sure the testing was done properly, so he was mostly in his test assurance role. Of course, the test coach also did some testing himself—mostly because he just likes testing, to be honest. And yes, with his tester’s eye he saw some defects the users missed.

With this combination of tasks, test assurance, and test coaching, the test consultant made sure testing was done professionally by nonprofessional testers. The users were involved early in the project and learned to do activities they had never really done before on a professional level.

The project was successful from a testing and quality point of view. During testing the users found an unexpectedly high number of 123 defects, and the system went into production without any major defects. It was also successful from a project management point of view because the timelines were met and the project was delivered after an intensive period of conversion testing.

Case 2

Case 2 is a large, limited company governed by public shareholding running a project in which twelve teams worked together. This project was part of a program to realize a well-functioning society-critical system, as well as a state-of-the-art continuous integration and deployment platform.

Within the program distinctions are made between the application level, performance level, and infrastructure level, and there are clear requirements for each. The project in this case is focused on the realization of the infrastructure. In this project SAFe was used to coordinate all work done by the different teams.

Most of the testing was done in the teams—engineers tested their own work as well as the work done by other engineers. But on a project level, there was no information about whether the testing done in the teams was done professionally. Making this determination was the first task of the test assurance officer and test coach—for practical reasons, we will call him a test manager in this case.

This was a highly informal assessment: When the test was not sufficient enough, the test manager came up with new test cases and helped the team optimize testing by focusing on the high-availability principles of continuity, availability, and stability. This was more of a coaching activity. Sometimes application testers (users who were not part of the teams) supported the test manager with this activity.

Although the testing done in the teams was important, it was not sufficient; a lot of the risks were related to the integration of the different infrastructure components. This system integration testing focused mainly on the high-availability aspects and on the integration of different infrastructure components, applications, and interfaces. The integration with existing systems (both applicative and infrastructure) was tested here, too. Different organizations were involved in this test. The architects were important in making the high-level and low-level designs and in guiding the architecture teams with a framework.

In supporting the engineers by means of coaching, the test manager sometimes crossed the line between test assurance and coaching on one side and test implementation on the other side. In those situations the test manager made the test design and test cases. It worked well in this project, but you should be aware that this made him less independent in his role. A mitigating measure was that the application testers were involved in making the test design and test cases, so other well-informed project members were involved besides the architect and the test manager. This lowered the risk of bad testing.

By the way, in this project the different teams and the integration testers made professional regression test sets. The sets were always reviewed, sometimes by the test manager, sometimes by the users or architects. Most of the regression tests were automated in order to make it possible to execute them fast.

At the end of the day the test manager, in the role of test assurance officer and test coach, had an end-to-end overview of what was tested where, if all the components as well as the integrations were tested, if the testing was done professionally, and if there were any blind spots. He couldn’t do this alone, so the test manager involved different stakeholders, both from the project as well as from the users. This way he made sure the infrastructure was tested thoroughly.

An Evolving Role for Modern Testing

We rarely see anymore the traditional test manager role: a project member on the management level who defines the test strategy, makes a test plan, manages those doing the testing, and reports the outcome. Most of the traditional test management activities are done by the team.

Figure 1 gives insight into whether traditional test management activities are not done anymore, done by the team, or done somewhere else in the organization. This matrix will be different for every organization, but this one was made at the conference SEETEST Sofia 2017, during the workshop Agile Test Management.

Because testing in modern projects is not always done by professional testers, this can introduce risks, especially when we talk about big or complex systems. Test assurance or coaching can be a mitigation measure to reduce this risk.

This is just one way that test assurance and test coaching, combined or separately, can add value to organizations, projects, and teams.