When you do the same thing many times, you can start to make false assumptions about your work process—and testing is no exception. Sofía Palamarchuk discusses some common fallacies about performance tests specifically, and how they can end up costing testers and developers significantly more than they should.
It’s always interesting to find out the many ways in which we can be wrong. In his book Perfect Software and Other Illusions about Testing, Jerry Weinberg explained a number of fallacies regarding testing in general. Here, I’m going to discuss some that relate specifically to performance tests—and how they can end up costing testers and developers significantly more money down the line.
1. The Planning Fallacy
We often think that performance tests only take place at the end of a development project, just before rollout, in case we need to do some fine-tuning to make sure everything goes smoothly. That’s why performance testing is seen as a solution to performance problems. But, in fact, it’s about detecting and anticipating problems in order to start working on their solutions. The problem is that when we consider performance testing only at the end of the project, when we encounter very serious problems, at that point their solutions come with higher costs.
So, to keep costs low, it’s best to consider performance from the early stages of development. We should carry out intermediate tests throughout the development lifecycle in order to detect important problems that arise before they spiral out of control.
2. The “Just Add More Hardware” Fallacy
It’s typical to hear that performance testing is not necessary because any problems detected may be solved by simply adding more hardware, such as additional servers, memory, etc. This assumption is quite mistaken. Consider the case of a memory leak. If we add more memory, we might keep the server active for five hours instead of three, but we won’t be solving the problem. It also doesn’t make any sense to increase infrastructure costs when we can be more effective with what we already have and reduce fixed costs in the long run.
In short, adding more hardware is not a good substitute for performance testing. Instead, we should be finding the root of the problem and applying a real solution instead of a patch.
3. The Testing Environment Fallacy
There is another hardware fallacy asserting that we can perform tests in an environment that does not resemble the actual production environment—for example, testing for a client on Windows and assuming that the application will function just as well for another client who will install the system in Linux. We must make sure to test in an environment as similar to the production environment as possible because there are many elements from the environment that affect a system’s performance, such as hardware components, operating system settings, and the rest of the applications executed at the same time.
Even the database is an important aspect of the performance testing environment. Some think performance tests may be carried out with a test database, but then problems with SQL queries might go unnoticed. As a result, if we have a database with thousands of records, the SQL response time will not be optimized and would surely bring along tremendous issues.