Step 1: Define Initial Requirements
Even before you begin looking at the tools on the market, it's important to
create a list of the initial requirements. What problems will the tool you
choose help you solve? What capabilities will the tool need in order to be
effective in your environment? What constraints, budgetary or otherwise, will
the tool need to meet? The initial requirements give you a starting place; as
you learn more about the testing tools available, you can refine your list.
Compatibility Issues
Any testing tool you choose will need to be compatible with:
- The operating system(s) your application supports
- The development environment(s) used to create your application
- Third party software with which your application integrates
For example, if your application runs on both Mac and Windows, you may want
to find a testing tool that supports both platforms. If your application is
written in Java, the tool you choose will need to support Java. If you are
testing a database application, you may want a testing tool that can retrieve
data from and possibly insert data into the database without using the user
interface.
If you are evaluating test automation tools, a particular issue to consider
is how the tool handles "custom controls" in your application. Some
GUI (graphical user interface) builders create custom controls instead of using
standard Windows controls. You may not be able to tell whether your application
has standard or custom controls until you do a hands-on evaluation of the
automated testing tool. However, if you suspect that your application contains
many custom controls, include compatibility with custom controls on your initial
requirements list.
Tool Audience
In defining the initial list of requirements, keep the audience for the tool
in mind. Ask questions such as:
- Who will be using the tool on a day-to-day basis?
- Is your organization willing to invest time and energy in training?
- Does your organization expect test automation to pay off right away?
The skill level of the people who will be using the tool will play a part in
determining the best tool for your organization. If your organization expects to
put the tool into the hands of extremely technical, senior staffers, then power
and flexibility will be more important than ease of use. If junior testers or
non-technical staff members will be using the tool, then ease of use becomes a
critical consideration.
In general, powerful tools have higher potential benefits if used effectively
and are correspondingly more expensive. But that power can be a liability if the
people using the tool every day find the tool overly complex or intimidating.
It may seem obvious that you need to consult the people who will be using the
tool on a daily basis about their tool requirements. However, when you’re in
the middle of an evaluation, flooded with information and pressed for time, it
is easy to forget to involve the users, especially if you don’t work closely
with all of them. Involving the people who will actually be using the tool in
the evaluation and decision process is critical to your organization’s success
with the tool.
The opposite problem of involving too few people is involving too many. It
may be very tempting to involve everyone in the organization at this stage.
Involve people affected by the tool but not using it on a regular basis as early
as it makes sense, but no earlier. Other people are likely to ask questions such
as, "how will the tool affect me?" and "can we find a tool that
can also solve this other problem?" You won’t know the answers to these
questions until you’ve done the initial investigation. You also don’t want
to expand the scope of your requirements so much that no tool could possibly
meet them. The best people to talk to now are the primary users of the tool. It may be premature to involve everyone in gathering the initial
requirements.
Budget Constraints
You'll also want to determine the budget for the testing tool before you go
shopping. Cost is likely to be a factor, and how much your organization is
willing to spend could have a significant impact on your choice of tools.
Remember that the test tool budget must take not only licensing costs into
account, but training and implementation costs as well .
As an example of an implementation cost, consider a test automation tool. If
the tool will be used for unattended test execution, the budget will need to
include funds for additional machines on which to run the completed scripts.
Initially, scripts won't take long to run. However, when you have enough scripts
to run for 15 hours, you won't want to tie up the computer on which you do most
of your work while the scripts execute.
When you and your management set the initial automated test tool budget,
consider these questions:
- How much can we afford?
- How much is this worth?
How much you can afford is a different question from how much a tool is
worth. If the future of the company rides on the ability of a given application
to perform well under load, then a good load testing tool may be worth a
tremendous amount to the company. Yet if the company is a small startup, it may
be unable to afford a high-end performance tool. On the other hand, the company
may have a large tools budget, but does not see worth in a given type of tool
because the development group has already created a custom tool that does 50% of
the job at 5% of the cost.
You will also want to consider these questions:
- What benefit does the organization hope to get from the tool?
- Is the benefit quantifiable and measurable?
Addressing the benefits as well as the cost in the budgeting process enables
you to discuss preliminary ROI (return on investment) projections. Even if you
aren’t thinking about ROI, your executives probably are. You’ll want to be
able to quantify the expected return on investment to ensure executive-level
support.
While considering the benefits of the tool, you’ll need to determine how
your organization will measure its success with the tool. Will you measure the
number of hours that the tool saves? The number of defects the tool finds?
Improvements in quality?
You may find that it is difficult to gather meaningful or accurate numbers to
quantify success. Choose your measures accordingly. Also, don’t be afraid to
ask for changes to existing processes or tools, such as new fields in the defect
tracking system, to make it possible to gather additional measures.
Business Requirements
If you work in a large company or regulated industry, your organization may
have additional requirements that tool vendors must meet in order to become
"approved suppliers." This is the time to determine those requirements
so you can eliminate unqualified vendors before you spend much time evaluating
their offerings.
Step 2: Investigate Your Options
When you are investigating your options, you want to spread your search net
as far and wide as possible to find the tool that's right for your organization.
Read newsgroups, search the Web, visit websites, attend tradeshows and
conferences, ask questions on related mailing lists, check out ads in industry
trade magazines, ask coworkers about tools they've seen or heard about, and call
vendors to get them to send you product literature. If you work for a large
company, ask other divisions what they use—they may already have a solution in
place.
In this stage, you’ll want to talk with the tool sales people. It is to
your advantage to give the sales person as much information as possible about
your decision criteria. Be very open with them about your requirements and about
their competition.
Ask the sales representatives up front if their tool meets the requirements
you have gathered so far. It's very likely that the sales person will respond
positively. The important part of this conversation is not when the sales person
says, "we can do that;" it is when the sales person says "and
this is how."
It is also reasonable to ask the sales person how their tool compares to
their competition:
- How can I compare your product with other products on the market?
- Under what situations is your tool the best choice?
- Under what situations is your tool probably not the best choice?
- What features differentiate your tool from the competition?
- What don't you like about your tool?
The sales representative may pressure you at this stage to do an evaluation
or to let them come in to do a demo. Resist this pressure. During this stage
you're gathering information that will allow you to decide which products make
sense to bring in for a demo. Demos and evaluations are time-consuming. Unless
you have a lot of time to spare, you want to avoid doing demos on products that
don't come close to meeting your needs. You want to avoid evaluating
products that don’t come close to meeting your needs.
Step 3: Refine Your Requirements
While investigating the tools on the market, you probably discovered
additional requirements you either didn't know about when you created your
initial list or that you forgot to include. Add them now.
During this step, you need to communicate the effects of implementing these
tools to everyone affected by them. Don’t just talk to those who will be using
the tools on a day-to-day basis. There are two key reasons to involve everyone
affected proactively: it gives them a chance to discuss their concerns about how
the tool will change their job, and their questions may imply additional
requirements.
Some people may have concerns that seem to have nothing to do with the tool.
For example, "If we automate everything, will I still have a job?" If
people are very concerned about or even threatened by change, then your
organization may have an additional requirement that the tool not necessitate a
drastic change in the way the department does business now. You may also choose
to address these concerns through training and communication.
Other questions imply functional requirements. For example, a manual tester
may ask, "How will I know which of my tests have been automated?" In
this example, the manual tester needs to know which tests have been automated so
he or she knows which tests not to execute anymore. The tool needs to provide a
method of tracking the manual tests represented by each automated script.
In your initial investigation of the tools on the market, you may have found
that the budget you set in Step 1 is unrealistic. The tools that meet your
functional requirements may not meet your budget requirement. If that is the
case, now is the time to make the business case for increasing the budget. If
management is unwilling to invest more than originally budgeted, you may need to
pare down the functional requirements.
Step 4: Narrow the List
Now it's time to go shopping in earnest.
It is certainly possible that there is only one tool on the market that meets
your requirements. If that is the case, this step is easy. (You will still do
the next step: even with a single candidate, you'll want to make sure that the
product can do what you need it to do before investing.) It is more likely that
there are four or five real possibilities and several more partial solutions.
Use your list of requirements to rank the tools you investigated earlier. The
top two or three from this ranking will be the final candidates that you bring
in for a hands-on evaluation. You may find that none of the tools meet all of
your requirements, but several of the tools meet some percentage of them. In
this case, it is a good idea to get input from others in your organization on
prioritizing the requirements so you can rank the possible tools definitively.
(See the side bar for a step-by-step prioritization process.)
Why choose just three? Why not bring all the tools in so that you can try
them out? The answer is that hands-on evaluations take a significant amount of
time. You probably don't have a lot of time to spend on the evaluations, and it
is much better to evaluate your top three choices thoroughly than to evaluate a
wide variety of tools superficially. It is much better to evaluate your
top three choices thoroughly.
Step 5: Evaluate the Finalists
The first step in evaluating the finalists is to identify a representative
set of activities to accomplish with the tool during the evaluation. You'll want
to choose one or more tests that pose interesting challenges for the tools.
The next step is to contact the sales representative for each product. Ask
for an evaluation copy of the tool. You'll want to have access to each tool for
at least a week, and preferably a month, to evaluate its capabilities
thoroughly. Gaping holes in functionality show up right away, but little
annoyances that will drive you crazy over the long haul show up only after
extended use. Most of the tool vendors will allow you to evaluate their product
for a limited time (30 days or so). It is unlikely that the vendor will be
unwilling to let you evaluate the tool, particularly if you indicate that on
on-site evaluation is a requirement for the sale. Little annoyances
that will drive you crazy over the long haul show up only after extended use.
Some of the tool vendors require you to allow one of their staff to
participate in the evaluation. Vendor representatives may suggest that they can
take care of the setup and write a representative script to perform a test on
your application for you. If the vendor insists on doing the work for you, it is
worth your time to watch over their shoulder so that you understand both the
complexity involved in setting up the tool and how to use it effectively.
During the evaluation, it is a good idea to model the entire process that
your group intends to follow. For a GUI automation tool, that means you should:
- Create the test
- Check the test into source control
- Run the test
- Evaluate the results
- Pretend the user interface changed and you need to modify the test
accordingly
Imagine doing the above with not one, but hundreds of tests. Does the process
scale? If possible, perform these steps several times, taking note of the things
you have to repeat. Then imagine how you would feel repeating those things 100
times.
Now That You’ve Decided
Congratulations, you've chosen a tool! Whether you or your management
negotiates the final deal with the tool vendor, remember that many vendors offer
quantity discounts. Consider options such as rollout plans in which your
organization commits to purchasing a given number of licenses over a specified
period of time.
When rolling out the new tool to your organization, remember to:
- Make time for training
- Take implementation one step at a time: apply the tool to a single problem
rather than trying to apply it to all problems at once
- Immediately begin taking measures that will tell you how well your
organization is succeeding with the tool so you can see progress over time
- Communicate clearly, often, and in as many different ways as possible with
all of the people in your organization affected by this new tool.
Finally, remember to go back to your list of requirements every now and then.
Are there features you wish you had thought about when you were evaluating the
tool? It's a good idea to keep your requirements list updated on an ongoing
basis. That way you'll have the information about what you need already gathered
if you must choose another tool.
