Practical Performance Testing on Any Budget


Testing application performance prior to release is an essential part of managing risk in any software project. But the budget must be considered when talking performance testing; you want to know what it is going to cost to build and maintain a system that supports the project goals. However, there are ways to test the performance of your project while keeping the effort to a manageable set of tasks that get the job done without breaking the bank.

“Why is it so slow?”

“Is it doing anything?”

These are the types of painful questions you don’t want to hear after your software goes live. It means first impressions are already going down in flames and there is little or no time to save them.

Sites or applications that take a long time to respond or behave in a predictable manner are not likely to get a second look, much less gain a good reputation. This is why testing application performance prior to release is an essential part of managing risk in any software project. It’s also why performance testing should be considered when talking budget; managers and executives want to know what it is going to cost to build and maintain a system that supports the project goals. The licensing, additional modules, and extra hardware needed for performance testing can eat into the project budget quickly if left unchecked. However, there are ways to test the performance of your project while keeping the effort to a manageable set of basic tasks that get the job done without breaking the bank to get results.

Let’s look at the market for load-testing tools. Vendors love to wow you with demos, videos, and even on-site presentations that show how quick and easy it is to use their tools to get results. I am often left scratching my head at the end and asking, “OK, but who needs to load test the desktop calculator in Windows when I will likely be dealing with an n-tier system that has a presentation layer, business layer, and backend database?” Or, “Who is going to be writing scripts for performing searches with Google?” Promises of rapid results and ease of use are sure to fade when you create your first scenario and discover the real work to be done. Accounting for unique IDs, security tokens, and data preconditions are sure to keep you busy with any tool you choose. The short version: There is no silver bullet. We know that in the applications we test we will have to deal with security, preconditions for testing, data seeding, and configuration. So, how does one avoid the pitfalls of choosing a tool while making a good choice for their project?

The majority of systems being built today are n-tier systems that expose a platform-independent web service layer used by both the client and API programmers, and there are many tools on the market that handle generating loads for web services and HTTP-based messages. If you run a .NET shop, a lot of tools are built right into the development package, Visual Studio. Some MSDN subscriptions even allow you to use an unlimited number of virtual users just for being a subscriber, saving the cost of the tools and virtual users. This is the method I have used the most, and it is my personal “go-to” package for all HTTP and .NET web service testing. For Linux-based systems there are fewer boxed solutions, but there are plenty of free tools for testing and analysis.

You shouldn’t be expected to use freeware on a multimillion-dollar project, so let the budget fit the size of the project. It is going to make your organization—and you—feel a lot better if there is someone to call when things aren’t working as planned. In particular, support is certainly something you might need when the tool is behaving strangely or you need to do something more unique to your testing. That is the comfort you can get with a paid product. This is not to say that there aren’t effective free ways to get support. I always try to register an account on any technical forums where folks are sharing best practices and stories of their experiences with performance testing. And one should never underestimate the power of using Google to solve any and all problems. Become very skilled at asking the right questions and you are not likely to be let down often.

How you architect your load scenarios can save you money in licensing costs. If one of your scenarios is to execute something that takes ten seconds and then to wait out one hundred twenty seconds, there is a great opportunity for savings by quickening the pace of each virtual user instead of adding more users. Make one virtual user do the work of two by shortening the wait time to sixty seconds and you can cut the number of needed virtual user licenses by half. At times, you may be able to emulate the work of five real users using a single virtual user. Look for periods where your virtual users are waiting and do some simple math around your scenarios to save.

One of the places you will be really challenged is when testing a legacy system—legacy meaning, well, old. I worked on a project that used a VB6 client that communicated with the back end over DCOM. I could not find a tool that would work with this thing. We could not get a successful proof of concept going, even with top-tier multiprotocol products such as LoadRunner. (I think the boss was happier that we didn’t, as it would have cost well over two hundred fifty thousand dollars for the solution.) We ended up using a bitmap-based load test tool called AppLoader that went through the client at a fraction of the cost. The downside to a GUI-based tool like this is that it is very resource-intensive because it is using full desktop sessions for each user. Also, your application client needs to be able to run multiple instances on a single machine the way the tool wants. However, when there is no access to subject matter experts and no way to split the product into layers, this may be one of the few ways you can go. Make every attempt to work with developers on options in this case when possible.

Many vendors will try to sell you on their backend monitoring solutions as extra “modules” or “packages.” These can cost thousands to tens of thousands of dollars more just to monitor your performance counters. You do not have to do this. In a Windows-based environment, you can create data collectors within the Windows machine and monitor that way.

Exploration into Windows Management Instrumentation (WMI) can help you create your own performance monitoring solutions for Windows as well. WMI is a powerful way to access almost any information about the health of a Windows-based system. The Bytes IT community has an introduction to the concept, and there are some great examples online to get you started if you simply search.

For Linux machines, there are other benchmarking and monitoring options to help round out your performance testing abilities.

Another option is to roll your own tools. This sounds like a major undertaking, but when you consider what the load test software is doing the majority of the time, the mist clears and it is very possible. After all, load test tools simply spin up a number of virtual users (worker threads) and attempt to use the resources of the host machine to simulate the traffic. You will have to keep in mind the specifics of your system, but it is likely that developers who wrote the client code are able to see how to simulate the client traffic. Then you just need a way to control a multithreaded running of the scenarios. I have successfully done this on some projects, and we ended up spending a bit of time creating a Windows application capable of starting and stopping the traffic. I wrote this application myself after some brief study on worker threads and HTTP request processing. We complemented it with performance monitoring from Visual Studio Test Edition, which is free to use to monitor performance for many systems at once and overlay data, and we were off and running. We were able to simulate five thousand users running on three quad-core servers to prove a desktop monitoring product quite accurately. The result was a trouble-free launch of the application to the enterprise. I think I earned my annual salary with that little idea.

It pays to be educated in your choices when it comes to performance testing. A good, solid web test tool will get you through the majority of today’s performance test challenges. Backend monitoring packages are helpful but often not worth the expense, and there are other monitoring solutions that are either free or low-cost that fit almost any project budget. If you have a legacy system needing performance testing, take special care to explore solutions specific to the product and to get solid proofs of concept before diving into the licensing and purchasing of the software. Finally, don’t be afraid to engage developers and explore the options of creating internal load testing solutions. It will pay huge dividends in the long run. If you are interested in that route, study topics such as creating web requests and multithreading with worker threads to get going.

With these helpful tips, you should be able to guide a decent performance testing effort without swallowing the project budget. Bring the intelligence to the table and see your application to a successful release.

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.