In order to keep up with the ever-shorter release cycles that come with the adoption of agile software development, many development teams are embracing test automation as a means to continuously ensure that every software release conforms to the desired level of quality.

This is a significant shift from traditional software development practices, where testing was often stuck on at the end of the development process and seen as a process burden rather than a benefit. Testers working in an organization that has adopted agile software development, moved to a DevOps culture, and embraced continuous integration and continuous delivery are therefore required to have at least a basic level of understanding of how to effectively implement test automation as part of their daily activities.

Unfortunately, lots of test automation efforts in agile software development fail, or at least do not maximize their potential.

I'd like to explore what I think are the two most important reasons for test automation not living up to the expectations that testers and other stakeholders in the agile software development process have. Then, let’s look at strategies and tactics for how to avoid falling into these traps in order to succeed with test automation in an agile environment.

Unreasonable Expectations

The first of the two main reasons I see so many automation efforts fail to reach their potential is because of the unreasonable expectations that precede its implementation. Too many team leads, development and project managers, and C-level executives (although other roles are not innocent, either) see test automation as the one-stop solution to all of their testing bottlenecks.

However, reality has shown over and over again that:

• Implementing test automation takes time, effort, and specific skills
• Automation is an activity that supports testers, not replaces them
• Far from every activity related to testing can be automated

Yet there's still a widespread belief that test automation somehow, magically and with the press of a button, will perform all of the required testing for you. When, after a couple of months of diligently building and running tests, this notion turns out to be a fantasy, the people who have been working hard on getting the automation to work are often scapegoated, and sometimes even laid off.

In my opinion, the best way testers and automation engineers can deal with this problem is to think and communicate before acting. Make sure that all stakeholders are on the same level with regards to what you can reasonably expect from automation. Take a look at previous efforts, both within your organization and in the wider software testing and development community, and learn from those experiences.

What should work? What will likely not? Don't expect automation to be the magic bullet that will solve all of your testing problems.

Lack of Attention Dedicated to Automation

The other main cause of failing automation efforts is the lack of time that development teams (and, on a larger scale, whole organizations) allow for creating useful, maintainable, and effective automation solutions. Even though it should not be a surprise that implementing automation takes time and effort, automation is still too often one of the first things to suffer when time becomes sparse.

This applies to projects, but it also applies to teams working in an agile environment. Even though automation is high on the wish list of many a software development team, when the end of a sprint comes close, delivering features almost always takes precedence over automation.

Note that I do not think that this is necessarily a bad thing. In the end, it's the features that deliver value to the end-user, not the automated tests that help safeguard them working correctly. In the long term, however, teams that allow the releasing of features to take precedence over everything might just run out of breath while trying to make deadline after deadline. It's almost as if they forget that adopting an agile way of working is about creating software at a sustainable pace while receiving feedback early and often, not about delivering features at warp speed.

Not allowing for enough time to create a solid automation solution also has an unwanted side effect: If you don't award automation the priority it deserves (and that may not be the highest priority), it is unlikely that your team members will have enough time to become skilled automation engineers. I see automation as a craft, and like any other craft, it requires continuous learning and honing of your skills.

Making Automation Part of Your Agile Development Process

Now that we’ve detailed two main reasons for failing automation efforts, I'd like to propose a step-by-step guide to help you avoid these pitfalls and successfully implement test automation as part of your agile software development activities. This is by no means a definitive guide, and not all steps might apply equally to your situation. But following them might just help you to be more successful in your agile automation efforts. 

1. Set reasonable expectations

As I said before, the success of any automation efforts starts with reasonable expectations. I find that asking and agreeing on the “why” is a good way to set these expectations. Why are we automating in the first place? Why do we think we need test automation at all?

Note that in my opinion, there are good and, well, less good answers to this question. “Because we want to be able to get fast feedback on every commit done by a developer” is a good reason, whereas “Because we don't want to have manual testing at all” is a prime example of a source of unreasonable expectations.

2. Treat test automation as software development

Make sure that all parties involved are aware that the introduction of test automation is essentially equal to introducing a software development project within your software development project.

This applies to both the project planning aspect (you should assign resources to it and allow for time spent on developing and maintaining the automation, etc.) as well as the technical implementation of it (you're writing code, so be sure to apply good development patterns and practices and respect the fact that test automation is a craft that requires specific skills).

3. Assign dedicated resources to automation

In order to be successful with test automation in your agile software development efforts, you'll need to make sure that the people responsible for creating and maintaining the automation both have the right skill set and get enough time to do so.

The number of people who will dedicate their time toward automation depends on a number of factors, including their skills, the type of automation that needs to be created, and the complexity of and risk associated with the application to be developed. If your organization currently does not employ enough people to fulfill your automation desires, or the people lack the required experience, temporarily hiring external experts might be an option worth considering to get you started.

4. Pick a starting point

Just like with any significantly large project, it might look like a daunting task to decide where to start with test automation. I have two pieces of advice here:

• Start either with some low-hanging fruit (this helps show stakeholders the added value of test automation quickly) or with a part of your application associated with the highest risk or the highest impact of defects.
• Try to avoid starting with end-to-end test automation, like using Selenium. While this might seem like the obvious choice when you're looking to write automated regression tests, this type of test is the hardest to write, the slowest to execute, and the most prone to fail, either due to changes in the application under test or to false positives caused by synchronization or environment (e.g., test data) issues. Instead, see if you can create a solid set of meaningful unit tests, or leverage API-level testing to verify critical business logic.

5. Make automation part of your definition of done

When you're working in an agile setting, it makes sense to make test automation an integral part of your definition of done for a given feature. Try to avoid these two traps, though:

• Including a statement like “All tests should be automated” or “We should have automation in place for every feature delivered”—sometimes automation doesn't make sense, is cumbersome, or is even downright impossible to create. Instead, opt for statements like “Existing automation is updated to reflect changes brought by this feature” or “Additional automation has been created where deemed relevant and useful by the development team.”
• Relying on percentages—“100 percent code coverage” is an empty statement. It says nothing about the quality or relevance of the tests. Likewise, “80 percent of all tests done are automated” does not make sense, either. For one, this relies on a one-to-one translation of tests executed to automated tests, an approach that has time and again proven to be ineffective. But more importantly, how do you define that 80 percent in the first place? Is that 80 percent of everything that can be automated or of all testing performed? I think you catch my drift here.

6. Learn and adjust

This should not come as a surprise anymore: Test automation is a software development activity, and when you're doing this in an agile way of working, it makes perfect sense to apply the agile principle of fast feedback, quick evaluation, and learning on the go.

You don't have to get it right from the start! Just like with your application under test, take the time to experiment, evaluate early and often, learn from any mistakes, and stick with what works. Over time, and with the right amount of nurturing, this should lead to an automation approach that fits hand-in-glove with your software development efforts.

Please note that every situation is different, and what works for one organization might not work as well somewhere else. Having said that, I sincerely believe that the information above will be helpful to most organizations that are struggling with effective testing and are therefore looking toward automation as a means of improving their agile testing efforts.