Has your team been on the search for a fully automated acceptance test? Before you set out on that adventure, check out some of the accomplishments and perils behind the quest for complete automation, as explained by Jeff Patton in this week's column. Fully automated acceptance tests may seem like the solution to many problems, but you should know that it comes with a few problems of its own.
In the years that I've been involved with agile development, I've noticed an ongoing crusade for a holy grail that teams have been striving to reach: fully automated acceptance tests. There are some technical reasons why this crusade is difficult, but more importantly there are good reasons why automating some types of tests too early can be a bad idea.
Unit Tests Verify Code Modules
Agile developers generally divide automated tests into two types: unit tests and acceptance tests.
It's becoming more common for developers to write unit tests as they write code. They usually write unit tests to the internal interfaces of the code modules being created. Writing test first helps to specify the module being developed. Once they've written the code and basic unit tests have passed, developers commonly beef up those test cases with boundary conditions, incorrect input, and error handling.
This practice generally works pretty well. When all developers write tests, most every line of code becomes supported by a body of automated unit tests that is executed against the entire code base. Running those tests regularly with every code check-in confirms that the code behaves as expected when the tests were written.
Unit Tests Help Avoid Bugs Getting into the Code Base
When a developer adds or changes an existing code module, he sometimes sees unexpected failures in test that seem unrelated to the code he's working on. If you're a developer, then these are the times when you probably just can't help grinning. You've been investing heavily in this unit-testing practice, and, in one quick run of a testing framework, that investment pays a dividend by showing the failing test and with it the approximate location of the bug you've unintentionally injected. The failed test in the module you weren't working on pointed out a code dependency you weren't aware of--one you can address before you check in the new code you've just written. Thus, you avoid injecting a bug that may or may not have been found later through other forms of testing.
Well-written unit tests cover a single module of code. Unit tests help with design changes or improvements to code by verifying that things still work correctly in all other parts of the code after the change is made. Unit tests are pretty easy to write. And, when making changes, the impact to the test code base is usually limited to the tests that verify the code you're changing.
But unit tests have their limits. They work at a granular level--code module by code module. They're not easily readable by business people and aren't written at a high enough level to test full business-process flows.
Acceptance Tests Often Verify System Use
In an ideal agile-development world, the person specifying the software would write acceptance tests as executable requirements, and running those acceptance tests once the software is built would confirm that the requirements were met. Automating all acceptance tests has been a goal for many teams for years now. Some have even succeeded.
But things start to get a bit sticky when tests are written to control the software from the user interface. These sorts of tests may start up the application, log in as a test user, and drive through a series of screens, entering information and pushing buttons just as a user would. They're amazing things to watch, and having a body of these tests exercising your system does give you confidence that the system does indeed work.
Acceptance Tests are More Expensive
These acceptance tests can be more time consuming and difficult to write. Because they simulate use,