Requirements Mapping Using Business Function Test Suites

[article]
Summary:
On this team, testers were overcommitted, avoidable defects were surfacing, and documentation was hard to find. Worse, trust and morale were low. Upgrading tools was out of the question, so the testers decided to take matters into their own hands and create incremental change themselves. Here's how a team added a new type of traceability to its requirement test case world.

When I joined the particular division where I work as a senior test manager, test maturity was low. Testers were endangering the success of iterations by overcommitting and not delivering on promises. Most testing activites were undocumented, and test execution was defined in an ad hoc way. Test engineers were overworked and had low morale—even the joy of a job well done was missing.

For understanding the situation better, we need to look at the organizational structure. Software engineers were grouped by product divisions. In a product division, multiple development teams worked toward common goals. This was our average team structure:

  • Development manager
    • Oversees team operation
    • Acts like a ScrumMaster
    • Is the line manager for developers
  • Senior developers
  • Product owner
  • Technical architect
  • Senior test engineers

The team was geared toward delivery—with the inherent mindset focused on functional delivery—but other layers of testing were casually missing, risking overall production safety. Test engineers, though reporting to the outsider test manager, had the responsibility to work with the assigned team, and always had the end result of a successful iteration as first priority. Possibilites of actual QA were limited, with no timeslots available to educate testers for a better outcome.

We created product development plans and updated them in Microsoft Team Foundation Server with the following structure: Large-scale activities would be captured as an epic, defined as around two to four months of work and consisting multiple development teams. Epics are broken down into features, which would be one to three months of work, with a limited number of teams involved. Features, then, would be broken down into requirements to be worked on by one team for one iteration lasting two weeks.

All these documents were captured as change requests. So, if Product 1.0 was designed, engineered, and delivered, later on, an epic and its features and requirements for product 1.1 only contained the changes required.

This fact by itself wouldn’t have been a problem, but test cases, if ever captured, were connected to a requirement. Once the requirement was closed, test cases also disappeared into a void. Due to badly tagged test cases, it was difficult to get any real understanding by creating queries alone. (Microsoft Test Manager, which stores test plans and results on Team Foundation Server, does not have a built-in test case repository. There can be a test case not visible on any UI that can be found only via query.)

If these weren’t enough challenges, we also had one particular misunderstanding of agile surface repeatedly: "We are agile, so there’s no need for test documentation."

A change was inevitable.

However, we had further stumbling blocks. Upgrading the available toolset or changing the methods, guided interactions, and management tools was out of question.

Because there were no other options on the table, my team decided we’d have to take small steps to effect long-term beneficial change in the organization.

To make the case for regression testing and the documentation needed for it, we added a new type of traceability to our requirement test case world. We created a new notion, the business function—not bound to epics, but to the currently available functions in a software application.

It’s a system of independent folders, each covering a specific business function. Ownership of this approach was granted to testers from day one, so we came up with some new rules, too. In order to support the business functions, we formalized two more requests: Please write down your test cases, and if ever anybody could reuse them, link them to a business function.

We named this system "business function test suites," or BFTS.

The long, sometimes painful process of habit formation had started, consisting of kind reminders to every test engineer (often daily) in order to establish a new routine. It only takes around five seconds to map a reusable business function, but it’s an easy step to forget.

We started to capture and map test cases against the new BFTS. After a couple of months, products began to receive version updates, and the testers began to realize the power of a neatly organized, clickable libary of test cases to facilitate their work.

Next to the test cases and their BFTS style of cataloging, we started to do high-level test designs where test engineers, after having received a requirement, reviewed it and wrote down informal sentences outlining a local testing goal. We also involved programmer peers in reviews, resulting in higher initial quality in test cases.

So, with the new methods in place, here was our improved workflow: When a test engineer receives a new requirement, they create a requirements-based test suite where a high-level test design is captured. During the high-level test design, the test engineer understands the BFTS entry point as a source for any test case to be reused, and as a destination where new test cases should be mapped. After finishing the iteration, we do a quick review of the test assets, finalize the mappings, and close down. 

With high-level test design and BFTS-driven test case implementation, our testers began to realize that a test case is their contribution to the overall success of their software. This system is their product, giving them trust in their collaboration and a real sense of ownership.

We’re able to understand our work better, as well as the time needed for design and implementation, and what test assets can be reused. All together, this raised the level of commitments, lowering risk and rebuilding trust. We have opened up ways of development by lowering the average effort needed to cover an iteration—from a QA point of view, freeing up time to be used learning and practicing test automation.

Two months were spent on collaborative problem analysis and understanding our technical possibilities, and nine months were needed to establish our new habits. After about two years, BFTS is autonomous, with around six thousand test cases, both manual and automated, mapped to its folders.

By taking control of our process, we have enabled test engineers to do their entire job in a better way.

User Comments

8 comments
Alisha Henderson's picture

Hi buddy, 

Hope you are doing great. 

Very nice article. 

And the way you have mentioned is awesome. I will surely follow this suggestion and guide my team also so that in future w edont face any problems related to mapping. 

Keep it up !!

January 4, 2018 - 5:44am
Balazs Schaffhauser's picture

Hi Alisha,

thanks a lot for your great feedback. It would be interesting to read about your progress

Cheers

January 7, 2018 - 5:19pm
Tim Thompson's picture

Great article! It shows that lack of test documentation is never a good idea. We use TFS/MTM and came across the same issues. We draw a big beneift from creating and using a test case library. We have tests for all UI controls and even entire sets of functionality that is reused in multiple areas. We either clone the tests, move them to the desired test suite, and finish up with minor edits, or we reference entire suites from the test case library. A list in a wiki makes it easy to check if all the standard tests we want to run on a page are included, especially for app security. We keep a reference to the test case library test case, which makes changing all like tests fairly easy.

January 6, 2018 - 7:34am
Balazs Schaffhauser's picture

Hi Tim,

thanks for the reply. It is nice to see others having similar issues, and fine solutions for them.

Cheers

January 7, 2018 - 5:21pm
John Daughety's picture

Hi Balasz, congratulations on a creative solution that makes the most of the tools you have at hand.  I find these are frequently the most enduring solutions because of the commitment from their designers, who are also the primary users.  I would bet it was a lot of fun to see the momentum increase as the value of your approach became more apparent.  I have a question for you about Team Foundation System, which my firm is considering for project management:  was it easy to implement your solution, and were you able to use Team Foundation System's reports to get to the repository of test cases easily afterward?  

January 8, 2018 - 7:37pm
Balazs Schaffhauser's picture

Hi John,
thanks for your kind words, yes, it was thrilling to see it learning to fly. I will never forget the first occasion, when a test engineer said, 'hey, it was really quick to find the needed old test cases'.
I have been implementing sw management systems, like TestDirector, SilkCentral, Jira and this time, some modification to TFS. Its templating system - if you have admin access to it, or influence over it, can be a bless, or a curse. Just like with anything else, its users must pay attention to details, and should not allow required practices fading away.
On BFTS
other management tools (mentioned above) are supporting (manual) testers' daily life better. On the other hand, the described modification was really quick to implement in a technical point of view.
On reporting from BFTS - if you add strict rules on test case management to the described method - meaning, the proper usage of planning state - Design / Ready / Closed, and the execution status, you can effectively use the built in charting system of TFS. Go to a speficic (or the topmost) folder of your BFTS tree, switch to the second tab of your folder, and add a chart of your wish to visualize your test asset state - I do like this one, as no external data queries and charts are needed. TFS can be used for most of the reporting needs.
As BFTS has been in the growing up state, pipeline focused development became more and more important, where testing of a software component or product became a quality gate in the pipeline - less and less reporting was used out of BFTS.

January 9, 2018 - 4:19pm
Balazs Schaffhauser's picture

thanks for making my article into the Top 10 most read of 2018.

January 3, 2019 - 6:06am
tomas Chang's picture

Thanks, good article! Indeed, the lack of test documentation is never a good idea. And many companies faced the same problems. I agree with Nick Benson that I saved you a lot of money and time. I want to recommend the company that is engaged in automated testing.

January 17, 2019 - 4:51am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.