Risk-based testing is an approach to testing that helps us handle our limited resources. It’s also a valid model for years to come because it focuses testing resources where they can have the most impact—regardless of whether limitations are due to budget, tight schedules, or even the uncertainty of an unexpected situation like COVID-19.
Let’s look at some practical tips and examples you can use to adopt risk-based testing in your organization.
Risk-Based Testing: The Basics
Every software project should establish objectives at the outset. They can include staying within a certain budget for development hours, achieving a projected defect level, or meeting a promised release date.
The risks of a software project can vary widely based on the type of feature or product being developed. The risk profile of medical device software is significantly different from that of an online shopping store, for example.
Risks, both internal and external, can affect your objectives. Unanticipated events can impose new risks on the project that might manifest in other ways. For instance, if a key developer decides to leave in the middle of a project, they might take with them knowledge that will take extra time to replace.
Risks also will vary depending on the type of development. If the code is complex, related to a critical component of the software, or first of its kind, it will carry more risk. If the project is altering noncritical functions, involves well-understood code, or benefits from experienced developers, the risks will be lower.
COVID-19 is an interesting risk to the business to consider. Prior to December 2019, it was largely unknown and wouldn’t have been a factor in any software project. However, as the virus has moved into a pandemic state, it is now something that must be considered when planning a project’s budget and schedule. Will it impact a team’s ability to work in the office? Will it infect members of the team, causing them to miss time? Planning for such an unknown is difficult, but when risks of this magnitude surface, they have the potential to significantly affect your team’s ability to meet its objectives.
Finally, it’s important to only consider those risks relevant to the software project. It’s easy to get caught up in one’s imagination, including risks that are either highly unlikely or only remotely related to the software project. Some risks are just too hard to predict or too unrelated to your project to add value to the risk-based planning project.
What Is Risk-Based Testing?
Risk-based testing (RBT) is a testing methodology that uses risk assessments to help allocate testing resources. The purpose is to efficiently assign testing resources to the areas that pose the greatest risks to the organization. I used the word “organization” purposefully, to make clear that we are concerned about the impacts of poor quality on the support team, customer satisfaction, and the brand.
RBT needs to consider two primary aspects: the likelihood of a bug occurring in the code, and the impact that a bug would have on the organization.
Codebases are rarely homogeneous. Applications are built by assembling code from many contributors of various expertise and experience. Open-source libraries may be included. Some of the codebase may be more complicated or bespoke than the rest. All of these factors influence the likelihood of a bug occuring.
From an impact perspective, how likely will a bug impact a user’s experience? Will that impact be a nuisance or keep them from completing an important task? Is the user likely to call support and raise an issue that requires manager escalation, developer disruption, and a quick fix?
Use RBT to help focus most of your efforts on the areas of your application that are more likely to have a bug that creates a big impact.
RBT involves assessing risks and prioritizing tests based on the risk factor of each test. The risk factor is the product of the probability of a risk occurring and its likely impact.
But how can we come up with those values? Here are three criteria to consider in order to assign a risk factor to a particular risk.
Criticality refers to the measure of the impact of a software defect. Criticality is far from being evenly distributed across a codebase. Most applications are likely to possess a core of high-critical code that the rest of the application depends on. If the code doesn’t function well, it will impact multiple areas of the application and likely produce defects. A critical defect might cause data loss, sensitive data exposure, and perhaps even threaten lives if it relates to safety-critical software.
Criticality can be helpful when calculating the impact component of the risk factor. If a bug happens in the most critical area of the application, the consequences will be far more serious than if it had happened in some area that just contains utility code.
In our test automation application, we depend on our algorithms to identify web elements. If the algorithm and related core code can’t identify web elements, our customers’ UI tests will erroneously fail, causing confusion, rework, support calls, and a loss of confidence in our solution. Worse yet, if our application produces false positives, it can stop the CI/CD process and prevent a valid application from deploying.
Here, churn means the number of changes a given file or module undergoes. An area of the application that developers change more often is more likely to have bugs than one they almost never touch.
Consider an e-commerce application that has been in production for several years. Its team likely makes frequent changes to the offerings, introduces new features, and tries experiments to increase purchases. The impacted parts of the code change often and thus are more likely to contain errors.
When churn is high, risk-based planning should propose additional testing—more unit tests, more integration tests, more UI tests, etc.
Finally, we have complexity, which is also an important component of the probability of finding a bug. Simply put, more complex code is more likely to have bugs. The opposite is also true: the simpler the code, the less likely it is to contain bugs.
The question then becomes how to measure code complexity. While there are many metrics you could use, you’d be hard-pressed to find one that beats cyclomatic complexity. In short, cyclomatic complexity is the number of possible paths in a method or function. That number determines the minimum number of test cases you need to test a function thoroughly. Additionally, code with high cyclomatic complexity may be harder to read and reason about, which can increase the number of bugs.
Now that we have a better understanding of RBT and the types of risks that can affect your software product, here are some steps to get started implementing RBT in your organization.
Start with a static risk assessment of your application. Create an inventory of your application’s major components and list them in a spreadsheet. Start with a manageable list of 10 to 15 key functions. Next, consider each area’s criticality, churn, and complexity as you assign the key functions a risk of high, medium, or low. Be careful not to get too granular in your initial assessments, as the goal of this exercise is to establish an initial baseline. Review and gain agreement with your team.
Next, map your test coverage against the risk assessments. You should have adequate test coverage for all of your high- and medium-risk areas. It’s important to note that a gap in coverage doesn’t imply which particular type of test is needed. The risk should be addressed with the appropriate type of test (i.e., unit, integration, or end-to-end).
Work with product management and development teams to better understand where new engineering is affecting key functions and how that impacts risk. The sprint project board (often a kanban board or similar management tool) should show the backlog and tasks that are currently under development.
Build your test plan, weighting more of the testing resources to the areas of highest risk. Newly developed functionality in high-risk areas will warrant more testing resources. In contrast, if the team is modifying functionality in a low-risk area of the application, your testing effort should be lower. If you have excess testing capacity, it can be allocated to build tests in undercovered, higher-risk areas.
Learn from your efforts, communicate with your team, and adjust plans accordingly. You may need to change your risk assessments because a senior developer moved to a new project, or your team is building something unique and complex. You also might realize that you can lower risks because you have built the test coverage you need.
With each passing year, our software projects seem to be getting more and more complex. At the same time, our days don’t seem to be getting more hours. Our teams aren’t getting new members out of thin air. In other words, our resources are scarce. The challenges we face are plenty.
That’s why, when it comes to software testing, we need a smarter approach than “just test everything.” Not all code is created equal, not all risks are equally likely, and not all bugs cause equal damage. Software organizations must take these factors into account to make better decisions on how to allocate resources and test more efficiently.
That, in essence, is what RBT is all about: testing smarter, not harder.