Testing in the Dark
Some test efforts are straightforward about the exact test cases that need to be developed and executed to verify proper functionality and identify any adverse effects. But requirements can only go so far in identifying areas to test. At times, due to the entangled mess of the software architecture, precise test case development and execution may be ambiguous. This is especially true of legacy software that has had numerous tweaks over the years.
In these circumstances testers may be given no information at all on the areas of impact, leaving it up to them to determine what to test.
Testing all varying combinations of inputs and state permutations with limited time and resources is impossible. More importantly, needing to perform directionless testing indicates a broken software development process.
Don’t accept the need to indiscriminately test with no clear understanding. Your testing should be targeted, and using a few practical techniques will focus your test effort where it matters.
First, focus on what you need to do now to fulfill the test directive. Then create a long-term strategy to prevent this predicament from happening in the future.
Short Term
Help is not a dirty word. Sometimes just asking for assistance from the developer can reveal areas to test.
In the absence of a meaningful response, take the opposite approach: ask what not to test. You may discover irrelevant tests that can be eliminated.
If these two options don’t provide enough information, try these tactics:
- If the developer has gone through a design and code review, ask the other participants about areas to focus on
- Identify optimal, reasonable, and unlikely scenarios through the application, focusing your testing on the optimal and reasonable paths
- Identify critical features
- Identify known problem areas that tend to break due to instability
- Determine functional dependencies between the code that is currently working and the new or fixed code, and test that functionality
- Share your test strategy and cases with the team (including what you’re not planning to test) and request feedback
- Set expectations about potential risks
- Test as early in the development cycle as possible
- Run your regression test suite
- Run automation where possible
Long Term
Dig the well before you get thirsty. There are steps you can take to reduce the possibility you will need to test aimlessly in the future:
- Hold architectural reviews to identify overly complicated design implementations, and campaign for their redesign
- Ensure developers who are inexperienced or unfamiliar with legacy software are mentored, guided, and held accountable
- Follow configuration management and deployment processes
- Use metrics on defect density per requirement to uncover unstable functionality, and push for its redesign
- Participate in design and code reviews to get an insider’s perspective
- Ensure developers are performing unit and integration testing
- Use root cause analyses to determine where problems originate, and be an advocate for correction
- Increase automation where possible
- Ask developers to identify areas of the software to test
Whining about the lack of test direction won’t let you achieve your objectives. Instead of cursing the darkness, light a candle. Precision test techniques will help you focus your test effort.
Richard, this is an excellent, well written article. It sounds like you have a lot of experience testing in the dark! So have I. I have employed many of your ideas myself. You have consolidated and summarized the ideas very well. Some organizations like the one I am in now do not have well written requirements. We do not utilize much automation, unit testing, or configuration management. Developers do not do code or design reviews until it is time for deployment when they are required to have documentation of a code review. So I too am left with very little to go by. I find myself asking the business analyst to clarify the vague requirements in the ticket she wrote and/or asking the developer what the code changes were and what feature(s) are impacted by the changes. And, promotion of updated code into production is scary. It is still being done manually, file by file, by the developers, so it is very error prone. I have to test the same thing three times. In QA, in pre-prod, and in production.
Hi Ernie.
Thanks for your comment. Sorry to hear of your testing predicament, I can empathize with you. Also, know that you are probably not alone in this situation. I'm happy to hear that you are proactive in attempting to work around the issues by communicating with the business analyst to clarify the vague requirements. It does sound like you have the right approach. It should be remembered that quality is everyone’s responsibility and not just the testers. Documenting your team’s inefficiencies, providing and presenting to management clear and concise solutions identifying time, quality and cost savings is another possibility. Doing something always solves more problems than saying something must be done. Even if not perfect, presenting something that is imperfect surpasses doing nothing flawlessly.
Best of luck!
Richard