How to Show Value When Building an Automated Test Framework

Here is the scenario. Your boss asks you to automate your manual regression test cases. You will need to buy or build an Automation framework and integrate it into your organization. And you will need to measure the ROI, because there will be additional costs for resources/SDETs, tools, and shifts in priorities of current tasks.

How do you begin?

Hopefully, this is good news for you and your team and a new challenge to embrace. This article will give specific steps necessary to execute the project and show value to the entire organization, especially your executive management.

First and foremost, start with a Test Strategy Document (TSD). This conveys professionalism and well informs others. This critical document describes your automated testing approach and informs the key stakeholders of your goals, which are:

1. To Inform
2. Gain agreement
3. Document your path forward

The main components of a TSD are:

• Scope & Overview
• Testing Approach
• Environments
• Tools
• Release Management
• Risk Analysis
• ROI
• Appendix

Let’s break these components down into their detailed parts.

Scope & Overview

Include General timelines, Methodologies, and Resources. Think of your methodologies as the higher-level guiding principles that help you define your approach. The framework is then more prescriptive and specific on your structures. Lastly, your processes, which actually implement your framework in a unified manner.

Testing Approach

Lay out your specific goals, the resources necessary, roles and responsibilities, and the types of tests you will undertake (e.g., regression, integration, system, performance, load, smoke).

Environments

Here, you define your environments, the purpose of each, and the expected pipeline flow. Standard environments include Development, QA/Test, Staging/Pre-prod, and Production.

Tools

Describe the tools and software necessary to either buy or build your framework (e.g., Selenium/Cucumber, JIRA). Also, define your standards, such as how you plan to write your test cases (e.g., BDD/Gherkin, User Story). Be clear about how you plan to create and manage your test cases, that is, your Test Case Management (TCM) system.

Release Management

In this section, describe your deployment pipeline and release management processes. Cover what repository you will use, your continuous integration and continuous deployment flows. etc.

Risk Analysis

The point is to identify and analyze potential future events that may adversely impact the team and the company as a whole. Do this to better understand what may occur, the downstream implications of those events, and what steps to take to mitigate or eliminate the risk. Areas to consider in your analysis should include high-value testing functions, code maintainability, frequency of changes, initial code triage, overall communication, and data.

Return on Investment (ROI)

ROI compares the monetary and other benefits of a program to the costs (monetary and time) of that program. When we look at this from the perspective of cost, we very often must focus on time benefits and the savings in resources, and the increase in productivity. For Testing, examples of metrics to focus on may be time savings using automation, time reduced in the QA phase of the life cycle, lower bug recidivism rates, increased number of test runs, expanded scale and coverage (cross browser/environment), and increased confidence and trust in deployed code.

Again, the importance of the Test Strategy Document is to convey:

• What will QA do
• Why QA will do it
• How QA will do it
• When QA will do it

This certainly lays the foundation and sets the expectations with your key stakeholders.

The next tool you can use to share and communicate value and progress is the Roadmap. The Roadmap helps align the Test Strategy and QA Team goals with the business. It identifies targeted initiatives (Epics) as well as detailed tasks and allows for assessment of progress along the way. It should remain fluid, and it is suggested you make it visual.

But a solid Roadmap is no good if nobody looks at it. Schedule a monthly review with executives and stakeholders to focus on progress against goals. In these reviews, be open to feedback and don’t be afraid to pivot. These meetings and the visual representation tend to be the best way to collaborate and manage shifting priorities. Be sure to record and version control any changes - you know, CYA.

While the Roadmap is high-level and good for the executives, you want to show value to the entire organization, including Engineers, Product and Project Managers, Analysts, and SMEs. You can accomplish this with a monthly review of Statistics and Accomplishments. Focus on stories (e.g., JIRA) and test case statistics.

Here are some tips for the larger monthly review sessions:

• Show what you caught early (what did not go to Prod).
• Show time spent in automation, and time doing other exploratory testing.
• Show regression test coverage and bugs found.
• Show bugs found and closed with recidivism rates.
• Show how QA supported the product in unplanned areas or interruptions (e.g., load tests, ad-hoc).
• Set up the meeting as monthly and recurring.
• Discuss the impact on the Roadmap, if necessary.
• Focus on trends…and not too many details. That is, don’t just show numbers…compare to the previous month’s data and over time.

To help prepare for these monthly meetings, create a task (in JIRA) for each month. This task is used to track hours and issues encountered daily. Within this task, cover work focus, accomplishments, new/updated test cases, interruptions, and ad-hoc issues. Then, take the information from the task and add bug statistics, test case execution, and automation runs.

In summary:

• Establish your foundation with the business and key stakeholders by writing a TSD.
• Build and maintain a Roadmap.
• Plan and execute structured monthly review sessions.
• Keep meetings to no more than 30 minutes.
• Show progress against defined goals.
• Focus on trends, not detailed numbers.
• Don’t be afraid to raise concerns and discuss pain points.

and…

It is okay to “toot your own horn.”

Then you too can show your value to the organization!