Recently I read a post on a mailing list about “failed test automation projects.” This expression made me stop and think. Automating different types of tests is an integral part of software development, and that’s not only on agile teams. I worked in a waterfall shop back in the ‘90s where the programmers automated all their unit tests and ran them in a continuous build. My test team, which was involved in every project from start to finish, automated GUI-level functional tests as well as performance and load tests. Our product had no critical bugs in production.
I’m dismayed that many people still think test automation is something to be done by a team of test automation specialists and that QA should be a separate team, even in agile projects. I think every software team would benefit if they stopped thinking in terms of “test automation projects.”
Why Do We Automate Tests?
We automate tests to help our team work at a sustainable pace with a manageable level of technical debt. We automate regression tests in order to shorten the feedback loop between check-ins of new code and discovery of regression failures. We automate load and performance tests because, well, there is really no other way to do those types of tests. We automate repetitive steps to save time in setting up scenarios for exploratory testing. We automate the generation of test data for various purposes.
These are not optional activities. They’re all necessary in order to deliver high-quality software that provides business value in a timely manner without killing the team through overwork or sapping them of the will to live by having them do the same tedious keystrokes over and over.
The Problem with “Projects”
Because the term “test automation” contains the word “test,” managers think testers should do the test automation. They purchase a GUI test tool, usually one with a “record and playback” feature, and start a project for the QA department to automate all the manual regression tests. Perhaps they hope that once these tests are automated, they won’t need that pesky QA department anymore!
Some companies form a team dedicated to the test automation project but staff it with testers. After all, they’ve purchased a test tool that they believe is so easy to use that anyone can create the automated scripts.
In these scenarios, if the teams actually automate any tests, they’re likely to do so at the GUI level. The testers who create the scripts most likely have little or no programming experience. They aren’t familiar with code design patterns such as “Don’t Repeat Yourself,” so the test code is full of duplication. The test tool may not enable good code design for the test scripts. The scripts are brittle and hard to maintain.
I’ve heard of test teams that had several hundred thousand lines of test scripts that no longer worked. I’ve met testers on other teams who spend 100 percent of their time updating tests to keep them passing in the continuous integration process. This is a bad investment that adds to technical debt and slows the development process down even more.
Some companies are wise enough to staff their test automation department with programmers. These have a better track record. However, if they don’t leverage tests to help guide development, as in test-driven development and specification by example (also known as acceptance-test-driven development), then they are missing out on significant value that tests could provide. And, if they don’t collaborate closely with the programmers who write production code, then they may have to work with code that isn’t designed for ease of test automation.
Leveraging Different Skills
Let’s contrast the “test automation project done by test teams” approach with a whole-team approach where automation is an integral part of software development.
Learning to automate tests of any type is hard. At first, it’s a lot of extra work with no reward. You only realize benefits if you do it properly and stick with it.
Over the past several years, I have asked many people whose teams successfully automate tests to share the key factors that enable their achievement. In every case, they credit a whole-team approach to overcoming impediments to automation. Programmers who write production code are, hopefully, skilled at applying good design patterns. Writing automated test code is generally easy for them.
Testers are skilled at knowing which tests to write and to automate. When a programmer and tester pair to automate a test, not only is it much faster than a tester’s struggling to automate alone, but also it enhances collaboration and communication that benefits the whole product. I personally enjoy automating tests, but I’ve had to admit that the tests I design on my own are not nearly as clear and maintainable as the ones I do in collaboration with my programmer teammates.
Database experts on the team help provide test data and procedures such as database refreshes. System administrators help integrate automated tests into continuous build processes. The diversity of skills on a software development team helps overcome any impediment to automating any given test.