Many people in the software industry who advocate for using open source tools cite the benefits of their being free, and of the ease of setup that allows people to switch between systems in case they decide the tool they currently have is not the best fit for their needs.
Those are valid points, but there are also many issues and drawbacks—specifically, the cost in hassle and in wasted time that comes from repeatedly switching systems.
As your testing team grows, your test management tool becomes an essential element for the organization and for the success of your processes. Today, with the complexity and the fast pace of most organizations in their product development practices, the costs of switching tools are usually very high and can cause substantial damage to any team, product, and company.
Just think about your last two or three projects and the one you are currently running. I am sure you are already working with a blend of manual testing, load testing, automation testing, and API testing; at the same time, you are surely trying to cover all the angles of your product, such as functionality, accessibility, and portability, not to mention the multiple platforms, devices, and versions that you need to verify.
I like calling this a reality of “controlled chaos,” meaning you are working simultaneously on a number of platforms, constantly evaluating risks, and defining on the run what areas of your product require additional testing time and efforts. It is controlled because you cannot afford to lose track of your process and coverage, but it is still chaotic, as you are working and getting feedback from many sources of information as they become available during this process of continuous discovery and testing.
This controlled chaos is one of the reasons for online visibility, and for systems that integrate all the different parts of your process while having testing information centralized in one location that is easily accessible to your project stakeholders.
Consequently, I think choosing the right test management tool is more important today than ever before, and the direct and indirect costs of switching tools are higher than they have ever been.
Choosing the Best Test Management Tool for Your Team
The simplest and more visible direct costs of choosing a tool are the cost of the software and the actual time it takes to migrate the process and the data to a new system.
The indirect costs may be more difficult to identify, but they are no less troubling. Among the things I have learned is that humans don’t like change, and learning a new system can take people time and a lot of mental effort. There is also the need to adapt to a new process. These things may seem small, but they can take a toll on productivity and can have a negative effect on your team’s already challenging schedules.
So, in order to ensure you only have to go through this transition period once, what are the main elements to consider when choosing the correct test management tool for your team?
Functionality: This is the most important aspect to evaluate. The main goal of a test management tool is to improve your everyday work and help your company’s stakeholders make better decisions.
When thinking about functionality around testing, you need to look for a system that will allow you to create and manage your tests—not only for your current release, but for the entire life of your project. You should be able to easily reuse your tests, group them based on your needs, and assign them to your team in order to be executed correctly.If you are looking for a test management system, you want it to integrate with the rest of the tools and systems you use as part of your project, including automation tools, bug trackers, CI tools, etc. You want to be able to integrate between isolated actions in order to generate a unified visibility point.
Even if some tools offer packages or an open source approach that allows you to modify and “write” the things you want in your solution, you still need to account for the time it will take you to write this extension for your system.
Deployment or Hosting Method: There are tools that provide software as a service–only models, whereas others also provide an on-premises version.
SaaS solutions have a number of benefits. Among them you have immediate access from anywhere in the world, usually a more affordable price, the ability to scale up or down quickly, and the ease of upgrades.
On the other hand, sometimes these systems can have some glitches and be less robust, meaning that you may not be able to reach the service at some times. This has happened to me personally with a CRM system that went down for six hours and left us basically blind until the vendor was able to bring the system back online.
Customization: You should be able to fit the system to your needs, not the other way around. Customizations usually come in the form of adding information via dedicated fields to reflect the different products your team may be testing, as well as the platforms you want to cover, the different customers you are working with, etc.
In my experience, customizations will change all the time and may even be different within the same company, so you need to be able to do this easily and on a project-by-project basis.
End-to-End Coverage and Integrations: As I mentioned before, testing today is more complex than ever, and it is usually done not only manually and not only by the testing team. Most companies spread their testing efforts over a number of automation tools in addition to the manual tests, as well as include in their coverage unit and integration tests that are run on external build or continuous integration platforms.
Your test management solution needs to know how to integrate with these external tools and platforms in order to provide an end-to-end and integrated view of the status of your tests.
Support: In a perfect world we would not need to get any help or ask any questions, but in reality, we will need assistance—whether it’s when you need to customize your system, you run into errors, or you want to integrate between the different tools in your process.
Support options can range from a community to email only to a live human chat, and even phone calls. Another important consideration is the availability of support. Some companies don’t provide 24/7 support, so it is important to check support hours and consider time zone differences.
Be Selective and Consider the Future
We have many systems out there that can help us manage our testing processes. But only a handful of them will meet the needs of your organization while keeping in mind that each organization may need a specialized tool.
You need to choose carefully and perform a thorough evaluation when selecting your test management tool. The fact that you can switch systems frequently does not mean you want to—or that it will not have a big cost to your process and to your team.
User Comments
Often times QA teams do not have the luxury to chose the tools that serve their needs best. So if you work in die hard Microsoft shop the QA tool will be Test Manager, period!
I happent to work in such a Microsoft only shop and found that Test Manager is the worst product in the VisualStudio ecosystem. It's slow, it's buggy, it lacks customization, it fills in fields that cannot easily be queried for...nevertheless, it beats the old crusty spreadsheets by quite a bit.
Totally agree, many times we have little to no say on what tools we get to use, and we should be grateful to have a tool at all.
Still, from my experience, if you can show value with an alternative tool, and you can also show that this tool integrates to the overall process, you may get management to agree on getting an alternative to what you have...
Don't lose faith!
My 2 cents.
Great article. Your readers might also find real user reviews for all the major test management tools on IT Central Station to be helpful.
According to the IT Central Station user community, HPE ALM is the #1 test management tool. You can see what users have to say about it here.
Thanks! I would add a remark, from an outsourcer's point of view. This problem of unification gets an additional level of difficulty when some development & testing tasks are outsourced to contractor team(s). Ideally, everything should be organized before adding 3rd party teams into the business scheme - then you just have to make them use the same tools as the core team. But not all companies are doing this properly.