How Continuous Testing Is Done in DevOps

DevOps does speed up your processes and make them more efficient, but companies must focus on quality as well as speed. QA should not live outside the DevOps environment; it should be a fundamental part. If your DevOps ambitions have started with only the development and operations teams, it’s not too late to loop in testing. You must integrate QA into the lifecycle in order to truly achieve DevOps benefits.

The adoption of DevOps practices is dramatically increasing throughout many different industries, mostly due to companies recognizing the numerous benefits DevOps is able to deliver.

DevOps does speed up your processes and make them more efficient, but companies that solely focus on speed and ignore quality aspects are likely to suffer a huge blow. Teams should focus first on quality by reducing defects and bugs before working on speeding up their operations.

DevOps is all about better enabling the software testing process to deliver quality results within a shorter time frame. Consequently, quality assurance is an important aspect of the DevOps methodology. Integrating QA within DevOps helps companies focus on giving their clients quality software before it ships.

Integrating QA within DevOps also plays a vital role in managing risks by ensuring the application is robust and stable throughout the development process. Continuous testing as part of a DevOps method helps detect bugs quickly, when they are easier and less expensive to fix, ensuring your application is fit for usage and enhancing a good user experience.

QA should not live outside the DevOps environment; it should be a fundamental part. But if your DevOps ambitions have started with only the development and operations teams, it’s not too late to loop in testing. You must integrate QA into the lifecycle in order to truly achieve DevOps benefits.

Why Should You Perform Continuous Testing In DevOps?

Software projects, websites, and applications are not static. They require regular updates and real-time changes to fulfill all the set requirements of the clients.

These changes used to be time-consuming and perilous, but now they can be attained more easily through continuous integration, continuous deployment, and continuous testing.

DevOps allows software testing teams to easily upgrade and deliver various products without interfering with their quality. That’s why most DevOps enterprises begin with the adoption of continuous integration practices: to ensure that everything works together.

Continuous testing involves testing a software application beginning in its early stages and automating testing throughout the development lifecycle. This helps the team examine the quality of the product at every stage of the continuous delivery process. And this process is not limited to only testers and developers; it also involves the contribution of stakeholders, operations, and even the customer. Continuous testing is an integral factor in the DevOps equation.

If continuous testing is performed properly, it delivers quick and uninterrupted insight into the quality of every new build of your software. This information can help you analyze whether the application is prepared to go through the delivery pipeline.

For example, when the code in a source code server like Jenkins is verified by developers, a set of automated unit tests is executed in the continuous process. If the tests don’t pass, the build will be rejected, and the developers will be notified about it. If the tests pass, the code will be sent to the QA servers for functional and load testing. Then those tests are executed in parallel, and if they pass the build, the application will be deployed in production.

Key Points for Continuous Testing Adoption

Before you begin implementing continuous testing in your team, there are a few points to keep in mind. 

The idea of continuous testing is to implement testing early in the software development lifecycle and at each branch involved in your CI/CD (continuous integration and delivery) pipeline. So before you transition a code change to the production environment, you should already have it validated at the staging ones.

This could be challenging, as you need to make sure that all your staging environments are exact replicas of your production. This requires more resources, bandwidth, and infrastructural cost. And even if you do have all the changes pushed to the staging environments, you can’t be sure about pushing them to production just because they worked in the staging environments, as your production web application will usually be facing a considerable amount of user interaction. The fact that there is more web traffic in your production environment compared to the stage environment is one of the common things testers often forget. If you are short on the resources and investment required to maintain different stage testing environments, then continuous testing is probably not a good idea for you.

You also must be ready with your tools arsenal. You need to have the right automation tools on board for effective implementation of continuous testing in DevOps.

This means you need to have the tools required for each layer of the test automation pyramid—UI testing, API testing, and unit testing—because you can’t expect one tool to cover everything for you. For example, if you are performing automated browser testing, you would require unit testing frameworks to validate your code changes at an earlier stage in the continuous deployment pipeline. However, at a later stage, you would need an end-to-end test automation framework.

In continuous testing, you are testing a single change on multiple test environments before deploying into production. A release may contain a bucket of feature changes, so you will be testing numerous code changes at multiple test environments, and for each code change you also need to perform regression testing on each stage environment. All of this could be time-consuming unless you know how to put parallel testing to best use. Many automation frameworks can execute multiple test scripts simultaneously, which will speed up the continuous testing through your CI/CD pipeline.

Also be sure to recognize false negatives and false positives. They are more common than you may know! When you execute a test automation script, it may show an execution error even if the system is working fine, or the automation testing script may show the test execution as successful even when the system met with errors. While either is dangerous, false positives can be more devastating because you believe everything is good to go and then you get an outage. 

Plan your rollbacks thoroughly. Even if you use all the best practices for continuous testing and CI/CD, there is no guarantee that the build won’t break in production. Always be ready for the worst scenarios. Make sure your data is backed up before you push changes to production. In case things go south, you can roll back to the previous production version quickly and perform a round of smoke testing to ensure the web application functions well again.

Keep in mind that when you roll back from the production environment, you can’t just validate your pre-production situation and commit the changes again. You need to evaluate the entire pipeline to make sure there are no loose ends, as well as all your test environments, because you don’t want to postpone the same release cycle twice.

Integrating QA and DevOps

The testing and technical teams should work hand in hand to ensure quality throughout every step in the software development lifecycle.

When integrating QA into DevOps practices—known by the term “QAOps”—the testing team should adapt and adopt certain quality processes:

  • They should strive to detect bugs at the earliest point in the development lifecycle and prevent potential bugs from reappearing in the production cycle
  • They are responsible for highlighting issues in the process and recommending necessary changes
  • They must ensure that all the environments required for testing are standardized with automated deployment
  • Apart from finding and preventing bugs, they should also focus on improving the overall quality of the product

Some people in the software industry think the requirement for QA is decreasing with the rise of DevOps due to automated processes. But that’s not completely true. Though automated testing techniques are advanced and fast, they also require continuous human intervention. Companies that lack enough QA professionals and resources are not likely not to achieve their customers’ requirements, mostly when it comes to updates and continuous changes.

DevOps was created to make developers think in synchronization with software testers, not to replace them. Software development processes are becoming faster through continuous deployment to meet ever-growing customer demands, and integrating QA with DevOps can help you fulfill all your objectives.

The integration of the QA process into DevOps also helps you manage various risks, making sure the end results are robust and more stable. QAOps is like a fitness regimen for software, as it makes it easy to detect bugs frequently and on time so you can tell when your applications are fit to run.

There are a couple of patterns for integration of QA into DevOps that teams can adopt:

  • Try conducting an exploratory test before any new feature is merged into the master codebase. The tryout tests are designed to ensure the system is adequately covered to deliver accurate results
  • Before you transmit a code change, make sure that your QA, developers, and other stakeholders involved in the continuous testing process are ready with their testing checklist. This checklist should include all the valid and invalid test scenarios they need to consider along with the expected behavior over specific test environments

Every business venture, company, and enterprise operates differently, so the ways of adopting QAOps may also differ. But the two suggestions above can be implemented across any software team.

You cannot deliver a comprehensive and quality service without a QA testing strategy, so QA is essential to the DevOps process. Integrating QA processes into DevOps operations helps both the testing and technical teams handle dynamic software environments and situations, deliver at speed throughout the CI/CD pipelines, and ensure the quality of the product—as well as customer satisfaction.

User Comments

Akshaya Choudhary's picture

Great article, Junaid. Those false positives and false negatives are especially tricky to detect. Integrating QA in DevOps is a necessity, so much so that I believe it should not even be given a separate name. Because, whatever we do, ultimately QA should be the defining gates for software to go into production.

February 21, 2020 - 5:50am
Junaid Ahmed's picture

Hi Akshaya, I am glad that you liked my article. Indeed QA are the gatekeepers of sending anything to production. That is why QAops is what I believe is going to be the next big thing.

March 2, 2020 - 1:39am
Patty Salmon's picture

I think Continuous Testing is one of the basics of an efficient development process and I'm really glad the whole Agile idea emphasized it. Between my teams we share our special CT kanban boards to maintain the communication flow and current information of the whole process (and its testing results of course). And even when the testing results aren't satisfacory, we are able to quite quickly change necessary elements and go further. More, I guess frustration level within my teams is a little lower thanks to it ;)

February 29, 2020 - 6:12am
Ksenia  Kunkevich's picture

It is of paramount importance for companies to place their concentration on both the issue of speed as well as the issue of quality. This means that quality assurance should never be neglected in regard to the environment of DevOps. But rather quality assurance should always be placed as a core integration when it comes to the environment of DevOps.

Maybe your implementation of DevOps commenced simply with teams for development and operations. If this is the case, the good news is that you can still incorporate quality assurance testing.

When you implement top quality assurance within the realm of the life cycle of development and operations, you will be able to achieve maximized benefits. Due to the reality that many changes are occurring within developments and operations in many types of industries, it is reasonable to see just how important quality assurance is.

Many companies are glad to now implement quality assurance in an effort to have maximized benefits delivered from their development and operations. Do not get left behind.

April 1, 2020 - 10:13am

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.