Requirements Management Is a Responsibility of QA

Lists of requirements rarely have items like "lack of bugs", or "does what it's supposed to do, and doesn't do what it's not supposed to do". (The latter is my definition of what testing is for.)

The problem is that the *requirements* are never quite done, not even after the software has been shipped. I've been in this business for more than 20 years, and I've yet to run across a non-trivial project that didn't follow this pattern: 1) you get "the requirements" from the user, 2) you design a system, during which you find the need for defining more requirements that weren't in the original list, 3) while writing code, still more issues come up that demand resolution with extremely specific answers. You wind up with requirements being defined all during the project, not just at the start. In a perfect world, the up-front requirements would contain all that stuff, but it never does. And it never will–like one quote I've squirreled away says: "software is an opinion about how a business works"

Snipped from feedback on an article in Stickyminds.com

The thrust of the article in Stickyminds ( “Start A Revolution” by Linda Hayes) was that the number one priority of the test team on a development project should be to verify the requirements, not find bugs. This article received some sympathetic feedback, but also some comments virulently opposed, as exemplified by the snippets above. The purpose of this paper is to expand upon Ms Hayes thesis, demonstrating the flaws in the negative feedback above.

First, it seems best to make some comments about definitions and roles. Many of the people most strongly opposed to the idea of testers examining requirements seem to be opposed to any change or expansion of the tester’s role. Having been around for a long time, I remember the days when “data processing” departments had business analysts, systems analysts, and programmers, and the role of the programmer was to produce code according to the specifications he was given. In the modern IT organization, there are some specialized or senior roles, such as DBA or Data Architect. But the basic role is “developer” or “software engineer”, and the people in these slots are expected to participate in a variety of system and program design activities, as well as hack out some Java or whatever. This is seen as perfectly natural – now. Twenty or thirty years ago, it wasn’t

Also noticeable in my list of roles above is that “tester” or “QA” or “QC” isn’t there. Programmers and users did the testing. Starting sometime around the 1970s, the virtues of having independent testers started to be recognized. Over the last twenty to twenty five years or so, it has also started to be realized that testing is a valid specialization (i.e. it is actually difficult), and that the whole subject of software quality is worthy of attention, and is related to the development process as a whole, rather than any single activity (such as testing).

This leads to the realization (or perhaps I mean it should lead to the realization) that for a non-trivial development project, there needs to be a staffing component for Quality Control (QC). One of the primary responsibilities of these people will be testing, but they should also address quality- and process-related activities throughout the project. In other words, your testers should be doing a lot more than just running tests. In fact, if you want a high-quality product, they have to. This is emphatically not a radical position statement. For a developer, the bread-and-butter is hacking out code, but they are also expected to develop use cases, design data structures, help inspect other developer’s code, etc. For QC, the bread-and-butter may be testing, but they are also expected to monitor the development process, to participate in (quite possibly as mediator) reviews of various work products. And they are also the natural candidates to handle the on-going process of requirements management (RM) during the project.

This is one of the flaws in the snippets quoted at the top of the paper. The criticism implies that Ms Hayes stated that requirements have to be complete before the rest of the development process can proceed (an old-fashioned “waterfall” life-cycle). In fact, nothing in her article states or even implies this. It is well known that requirements change, are refined, become obsolete, and that new requirements arise, during software construction. How do we know what system to build, if we don’t know what the user wants? Partly, the answer is also provided in the snippet. The developers elicit requirements as they deem necessary during construction. Partly, the answer is the unacknowledged consequence of this process: sometimes, projects build the wrong system. Notice that executing the code to find procedural bugs doesn't help in this situation. The delivered system can be (to take things to the ludicrous extreme) bug-free and completely useless. An analogy I read somewhere (probably in one of Boris Beizer’s books) is digging a hole. It doesn’t matter how perfectly shaped the hole is, or how smooth the sides are, if you dug in the wrong place.

There is another flaw in the snippet. Granted that there is an on-going RM process during a development project, who should be in control of it? Following the same reasoning that favors a test team independent of the developers, in order to obtain objective testing, it seems clear that the developers should not be running the RM process, for the same reason. As explained all too clearly in the snippet, the developers will largely seek clarification of existing requirements, or elicit missing requirements, only when prompted by technical considerations. What is needed is a group who will review requirements with a view to making sure that they are complete and testable, based upon a business requirements point of view. Once again it is a question of objectivity, in the sense of independence from the system design considerations.

Then the question becomes, why the QC team? Why should the testers look at requirements? The reason (amazingly enough, bearing in mind that both snippets come from the same piece of feedback) is given in the first snippet.

As explained above, you can build a system without requirements. What some people don’t seem to realize is, you can not test a system without requirements. Consider the mission of testing as defined in the first snippet.

"does what it's supposed to do, and doesn't do what it's not supposed to do". (The latter is my definition of what testing is for.)

How do you know what the system is supposed to do? How do you know what it is not supposed to do? The answer, of course, is that the requirements define these things.

If that seems a little trite, take it to a more detailed level. How do you define a test case? Well, you specify any preconditions, you specify the input data and the user actions, and then you specify the expected results. The execution of the test compares the actual results to the expected results, and if they are the same, the test case is passed. How do you know what the expected results should be? I'm afraid the answer comes out the same: the system requirements define how the system is expected to behave.

Part of the criticism in the snippets is that requirements documents and definitions are never so completely detailed that every detail of an expected outcome is specified. That is certainly true. It is equally true that all this means is that not all of the requirements are documented. The requirements are still there, and they are still what defines the test, they simply haven't been written down. One of the useful aspects of having the test team working on requirements is that they can address these implied requirements, deciding which are trivial enough to leave undocumented (or covered by generalized statements), and which need to be dragged into the open and examined in detail.

Another aspect is that QC, in some respects, acts as the advocate of the customer or end-user. It is therefore relatively natural for them to be looking at the system from a business requirements point of view. This makes them better placed than the developers to identify requirements that have been missed entirely, or design flaws like error messages being routed to the wrong group of users, etc. It also places them well for oversight where a complex control flow involves components being developed by different developers. Where there is some ambiguity of definition, it is quite possible to have two developers taking different interpretations and finishing up with an integration error where the components don’t work together. Yes, this can be found in integration testing, but it is distinctly more cost-effective to have someone reviewing the requirements for ambiguity and making sure that everyone agrees before the code is built.

So this is where we get to the point. The first responsibility of the test team is to manage the system requirements. Because the ultimate responsibility is to test, and you can’t test without requirements. (And in between, there is an implication that the testers will also have some level of responsibility in documentation reviews, design discussions, and any of the other fun activities that go into the software stew).

I have had some interesting discussions around this subject with developers and project managers. Often, the first reaction received is to assume that as senior QC person on the team, I am in some way trying to usurp the project manager’s role or authority. Actually, that's a definite no. In fact, once the project gets properly under way, the PM discovers that I become his right-hand, helping with some detail tasks and freeing him to concentrate on more strategic items. Another stumbling block is a simple definition of terms. For me, a requirement is anything that the system must or must not do. Some people manage to categorize things so that some requirements are not actually requirements. (The only reason I have discovered for this is so that they can pretend that they have all the requirements, because the things they don’t know are not "real" requirements). I have yet to encounter a situation where we couldn’t reach a reasonable definition of terms, roles and responsibilities, given enough hashing things out.

A consequence of all this, that is worth mentioning, relates to the experience and training necessary for the QC team. Just as developers need to know more than language syntax, testers need to know more than simply how to test. In fact, the senior QC staff should be among the most experienced and knowledgeable staff in the company. They have to be, because they need to understand a fair amount about the business (in order to adequately represent the user), to understand the software development process (in order to fulfill the QC tasks at different points in the SDLC), and to have at least some technical background (because the nitty-gritty of testing will inevitably demand it). They have to be able to communicate verbally and in writing with everyone from the CEO down to the newest recruit, on both the IT and the business side. In short, if you have Superman on your staff, the best position for him is probably head of QC. Part of the reaction that "testers can't do all that" is simply the recognition that the company hires staff who aren't qualified and calls them testers. How do I express this gently? The problem isn't the definition of tester. The problem is your hiring practices.