For organizations looking to deliver quality at speed, running automated tests is an important part of the software development lifecycle. Test automation, however, can only be successful if implemented properly. Discussions on what constitutes a “proper implementation” of test automation often focus on what tool should be used for the job, or on the best (if there even is such a thing) or most efficient way to use a specific tool for a given task.
In my opinion, though, the tool that is used is only one part of the total test automation equation. Any successful test automation implementation is constructed from five distinct parts.
In this article, we'll take a look at each of these parts, how they contribute to the success of your test automation implementation, and what risks are associated with failing to pay proper attention to each of them.
1. The Test Automation Tool
While not the only factor playing a role in successful test automation implementation, the tool obviously does have an impact on the overall outcome of your automation efforts. Choosing a tool that is insufficiently compatible with your application under test, or one that does not fit the skill set of your automation team, will likely lead to less than optimum results.
Even more important than the choice of a tool, however, is asking yourself what it is exactly that you want to cover with your automated test, and then deciding on the most efficient way of getting to that result. A prime example of a question that needs to be asked is on what level a certain piece of functionality or business logic needs to be verified.
Do you want to make sure that your customers can open your web shop, search for a specific product, and subsequently place and pay for an order? You will probably want to check this using an end-to-end user interface-driven test. If you're verifying the correctness of a piece of logic that determines whether or not a customer is allowed to purchase a given object (for example, due to regulations on the state or country level), then you will likely be able to write tests that hook into your application under test at a lower level, such as an API or even a single class of code. This constitutes a different scope and approach for the test and, as a result, requires a different tool.
In short, make sure that you first know what your automated tests need to verify before spending time on how to achieve the desired result. Remember that there's a significant risk in forcing your tool to do things it is not designed to do.
2. Test Data
Another important factor of any serious test automation solution is the approach taken to managing test data. The broader the scope of the tests, the more important, but also the more demanding, test data management becomes.
While in unit testing you can get away with mocking all data your tests depend on, when you start working on integration or end-to-end tests, you will need specific data to be present in your application under test. And, to make matters even more complex, you will often need the data in other systems that interact with your interconnected application under test to be in a certain state as well.
There are several ways to deal with test data in these types of tests:
- Creating the required test data in the setup phase of the test
- Querying the system for existing test data before starting the test
- Initializing the database of your application under test before the start of a test run
Each of these approaches has its potential pitfalls:
- Creating test data in the setup phase of a test increases test execution time, increases the risk of failure before the test itself is even started, and leads to a lot of useless test data if there is no proper data cleanup procedure
- When you query the system for existing test data before starting a test, you run the risks of accidentally using invalid test data, or of no test data with the right properties being present in the system
- Initializing the database before a test run leaves you with database snapshots to manage and keep up to date—that is, if you're even allowed to perform a database restore procedure in the first place