By now, you have probably seen the Computer Associates television commercial with the cardboard sales people "pedaling their wares." The cardboard sales rep asks the prospect how many copies he would like to purchase. The prospect says, "We'll need about twenty copies of your soft…." But the sales rep interrupts him and says "Great! Five-hundred it is!" As the customer tries to respond, the sales rep continually interrupts with "Great! Five-hundred it is!"

This frustration is especially true for those organizations in the process of selecting test automation software. It is as if software vendors do not listen to your needs. I find that most companies have a difficult time selecting the software that will actually meet their test automation needs. The problem seems to be not in finding out which tool is best, but how to put a process in place to actually make a sound decision. This article is to help organizations identify and choose the best test automation software based on their needs and not the vendors' needs. This process should include evaluation of all products: commercial, open source, and freeware test tools. How does a company go about choosing a test automation package? Here are a few of the most common flawed approaches made by various organizations.

Vendor Bake-Off
Most companies will usually contact several vendors of choice and have them all come in and do what is known as a proof-of-concept (POC). The POC is when the vendor validates his tool(s) against your application in your test environment. The one that works the best with the customer's application (and has the best price) usually wins the business. However, this is only a very small piece of the evaluation process. The key ingredient missing in this process is the criteria needed to properly evaluate a vendor and its software.

Dive Right In
Another method used is when the customer requests an evaluation copy of the software. Although this is not necessarily a bad method for tool selection, it can be frustrating. For example, a user downloads the software and decides to jump in and begin testing with the software right out of the box. People often install the software and start using it without reading instructions, online help, or product FAQs. Then the user may encounter problems and assume the product does not work. The user will spend a significant amount of time on the phone with technical support. Test tools are not always so easy to figure out. This usually ends in frustration. In addition, the user has the task of evaluating the software and doing his actual job (imagine that). Time is lost, and the user must request an extension on the evaluation license. Imagine if this is done with five different test tools.

The Official Test Tool Guru
The next method some companies use is to have someone within their company, who is an "expert" or who has familiarity with a particular brand of test tools, lead the evaluation process. This person may have extensive knowledge with a single tool, and may have never worked with any other tool on the market. They may have used this tool successfully with a previous employer. This can lead to a biased proof-of-concept rather than an objective one. The tester will naturally gravitate toward the tool with which they are most familiar. This could lead to the company purchasing a tool that does not meet the criteria of the organization. On the other hand, extensive tool knowledge is definitely an advantage. Test tool knowledge helps companies extract the "vendor fodder" and save time in the evaluation process. But that's only a small piece of the overall selection process.

Trial By Fire
The next method employed by some companies, is to have a novice group of business analyst/testers who currently do manual testing, dictate the evaluation. This is usually the group that heard through the grapevine that these tools will "make their job easier" and "reduce their testing time" with "no programming required." This can be very detrimental to the entire tool selection process. The tool that seems the least intimidating usually wins. Unfortunately, all of the testing tools require some sort of programming expertise. This does not mean that the tester must be a programming guru, but must have a basic understanding of programming concepts in order to develop and maintain their test code.

Trial By Committee
This is when a company uses several members, involved either directly or indirectly in the tool selection process. This may consist of employees from different departments (development, QA, test team, operations, support, business development, technical writing, etc.). Each committee member has a vote to decide which tool meets the organization's needs. This segment of the evaluation process has a distinct advantage because you have the expertise of several groups. But this also means that members involved in the selection process have a vote, but the tool chosen has no direct impact on them. For example, Operations may not care about test automation, but may have a need for production-monitoring software. Technical support may not care about performance testing, but may have some interest in defect tracking software. The committee should consist of a group of individuals who truly have a vested interest in the implementation of the test automation process. Bringing in members with whom you interface, but have no direct impact in your test automation strategy, could be detrimental to getting the right software.

The Process
How does one go about selecting the right automation strategy? Here is a process that I recommend:

Know your needs. Before you get on the phone and start calling vendors for evaluation copies, you should first take a close look at your organization and your testing needs. Is your organization truly ready to begin test automation? Who on the team is capable of using these tools? What are their skill sets? Are you looking for test automation, test management, performance testing, defect tracking, or all of the above? Do I need a committee? If so, who should be involved? How much do we have in our budget? Does our budget allow us to start a useful test automation strategy? What are our testing goals? These and several other questions will need to be addressed before the search begins.

Establish testing criteria. Knowing what technologies need to be tested is extremely important. What is my application platform (Java, .NET, HTML, ASP, JSP, Siebel, Oracle Forms, SAP, etc.)? If performance testing, what protocols are involved? What is the overall architecture of the system? What other projects and technologies will I have to support in the future? Make a list of the critical (and noncritical) technologies involved in the development of your application. This will help narrow the search if a particular vendor does not work (or work well) within your environment.

Do your research. Now it is time to research the various vendors and products that will best serve your testing needs. Find tool lists at sites like StickyMinds.com and qaforums.com. Check out various vendor websites that specialize in testing. In addition, use Google and other search engines to find test tools, and to search for open source alternatives. Again, never rule out a free alternative product if it can get the job done. This is where your test tool guru will come in handy. Utilize their expertise, and get their knowledge on their tool of choice. Also, investigate other divisions of your company. Your company may have already purchased a tool (or several tools) in another division for their project. It would be wise to get their feedback and experiences with various test automation software.

Create a Proof-of-Concept (POC) Checklist. This document will allow you to define what the tools will need to accomplish in order for your organization to successfully use the product. This document should detail your application, technologies, platforms, and what tasks need to be accomplished with the software before a purchase is considered. For example, you may have several test cases you would like to execute, which show the tool's ability to work with your application and test the business rules. Strong object recognition may also be a criterion. The POC checklist should contain everything you feel you need to see in order to make an informed buying decision. However, this list should be reasonable for all vendors and participants involved. Extensive POCs (evaluations in which the vendor participates over several weeks) may cause the vendor to charge you for time spent using their tool with your application. This is a very important step in the overall strategy. Sometimes the vendor will provide you with a template of a POC checklist. This should be used as a starting point, and not the official document. If you don't set your criteria, the vendor will do it for you. The tool needs to meet your requirements, not the other way around.

Narrow your search and contact potential vendors. You have done your research, and now you can begin to contact the vendors who seem to meet your criteria. This is where the steps above save you a ton of time. You must realize that when you call vendors, you will be in contact with their sales representatives. Sales reps are trying to make money. When a potential customer calls in stating that they are looking to implement a test automation solution, they will immediately want to provide you with a software solution. They will constantly call you trying to set up an onsite demo and/or POC. In order to save you and your organization the headache, proper planning is the key. Put yourself and your organization in a position in which you will be prepared to listen. Without proper research, all the tools will start to look alike after the onsite demos.

Bring 'em on site. The next step is to have the vendors come on site and complete the POC on your application based on your criteria. Make sure that you give the vendor a pristine workstation to install their application. This machine should have the proper amount of virtual memory and hard disk space. Also, make sure to have only one test tool on the workstation at a time. During the POC, some vendors will complain that a competitor's application is interfering with theirs. I have seen this happen many times, and most concerns are well founded. Each tool is doing something similar, chances are there is a conflict. To limit quirky behavior, provide a "clean" workstation for each vendor.

Establish selection criteria. What if two or more vendors meet your criteria? What will be your deciding factor(s)? Will it be price, ease-of-use, ease-of-maintenance, quality of service, licensing, company background, etc., or a combination of each? What if they are dead-even on price? Establish what you consider the differentiators in your selection process. Occasionally you will have at least two vendors that can do the job very well. However, you must decide which is best for you and your company.

Validate all that you will need to purchase. In other words, if Pie In the Sky Test Company completed the POC with several add-ons (web-testing, Oracle, SAP, etc.) Make sure this is noted with the price so you get exactly what you need to do your job. There have been times where a customer thinks he can test every platform known to man based on the "entry-level" price, but then realizes he is not licensed for the particular components needed to complete the task. This will also allow you to objectively compare prices from competing vendors.

Negotiate. If a vendor quotes you a price that you don't like, negotiate. If a vendor can get you to pay the original sticker price, he will. As the old saying goes: you get what you negotiate. Do not let price hinder your selection process. The ultimate goal is to purchase a software package that meets your testing needs and budget. Try to negotiate from a win-win perspective. Make sure your quote includes price ranges for the software selection process. This will be based on your allocated testing budget as well.

Communicate your service needs. Communication is the key to building a working relationship with your test automation vendor. They must be responsive to your needs, whether it is through technical support or sales. The service you receive should be the same whether you purchase a single copy or a hundred copies of their software. Don't settle for anything less.

There are many ways to evaluate and implement a test automation strategy. The six Ps still apply (I added a new one): Proper prior planning prevents poor performance. As testers, we sometimes forget that we must establish our requirements, just as we request it from our organizations. Remember, if you do not establish your selection criteria, the other vendors will do it for you.