If you’re like the customer I spoke to yesterday and it takes you twenty-four to forty-eight hours to set up and build every load testing environment, then your ability to test and deploy new builds is definitely constrained.

Building a load testing environment is a challenging task that can require resolving technological, organizational, and security issues. This article can serve as a roadmap for building a faster and more efficient load testing environment that leads to quicker deployments.

1. Use your production environment

Yes, this is a bit awkward as the first tip, and it is rarely feasible. Still, your production environment is the ideal environment for testing, too. There is no need for replications, and it always includes those background processes and environment variables that affect performance and are often missed.

A few of our customers use their live production environment to run load tests during nights and weekends. It’s definitely doable, and it is my first choice for load testing whenever possible.

When using a production environment for testing, make sure to:

• Back up production data so that you can quickly revert to the real environment once testing is done.
• Either isolate or disable third-party applications (such as payment systems). For example, when testing an e-commerce site, include Visa or PayPal payment activities and delete all test data. Alternatively, you can use stubs to simulate such activity. If this is also not possible, then run transactions only up to the point of the third-party activity you want to avoid, such as transferring money to an account.
• Use fictitious entities (such as user accounts) to simulate the activity you want to test.

2. Make an exact replica of your production environment

Extrapolating load test results is a very risky business. If you run the test on a one-gigabyte machine with two cores, what will results look like on a two-gigabyte machine with four cores? It's practically impossible to know.

Therefore, you should replicate your production environment in every aspect: machine profiles, configuration, database, network architecture, load balancers, firewalls, etc. One method is to create complete images of production machines to be duplicated in your test environment.

3. Replicate a customer environment

Let’s assume your product is an on-premises software—for instance, an information system for universities that manages students, registration, administration, etc. It would be quite difficult for your research and development or QA teams to simulate a real-life environment for testing purposes. The most effective way would be to duplicate a customer’s production environment.

Try working with select customers with the assurance that you’ll protect or remove critical data, and offer to provide discounts or free support. Essentially you’re testing your customer’s real environment and taking over some of their efforts, so it can work out to be a win-win situation.

4. Build a real environment from scratch

In some cases, you’ll have no production environment to copy and will have to build a load testing environment from scratch. You may be in the early stages of developing your software and don’t have a production environment or enough customers yet, or you may be restricted by security issues and have no access to the real production environment.

In such a case, make sure to:

• Take into account different types of potential deployments you might want to test. With each, address the number of users configured, how many records are in the database, etc.
• Create configurations as similar as possible to the typical production—security, hardware, software, network, other apps deployed, load balancing, and so on.
• Use automated scripts to populate the database representing the real environment.
• Incorporate less apparent ongoing procedures. Quite often, performance degradation occurs due to periodical tasks such as a database backup or a weekly business intelligence report generated for management. Problems are not detected because these periodical tasks were never configured in the test environment.
• Add third-party integrations used in the live production system, such as payment systems, reporting or support, etc.

5. Establish a procedure to quickly set up your test environment

You want to be able to set up your load testing environment as quickly as possible so that you can reuse it for new tests. Once you have your environment working and configured, create a snapshot or image so that you can recreate it in minutes. There also are many configuration management and IT automation tools you can use to quickly set up a server environment, such as Puppet, Chef, Saltstack, Ansible, Docker, and Vagrant.

6. Isolate the test environment

Computing resources are always scarce, and test machines are often the solution. When the CFO or the new product team is looking for a machine to run a process or trial, they’ll often land on some test machine—and don’t be disappointed if you’re not notified. You want to make sure that no one but your virtual clients can reach your tested application. It’s important that your test network is isolated so it is not affected by other company activities you may not be aware of.

Building a realistic test environment is essential for the success of your load testing. Initially, it may seem like it slows you down from launching your performance testing, but think of it as the structural design of a skyscraper. You simply cannot do without it, and it will eliminate the chances of catastrophic failures.