The Unspoken Requirement: Testing for Consistency

It's easy to see that style consistency is important when discussing the user interface. But there are other areas where being consistent is just as important, even though they are not as visible. Consistency is one of the quality attributes of a product—any product—even if it is not stated clearly in the requirements documents, and testers have a responsibility to check for it.

About ten years ago I was testing the installation package for a Windows application. The installer was a standard one, similar to what we are all used to: a set of screens through which most users will usually click Next, Next, Next to install the software.

As long as I was using the mouse, everything was fine. But because I prefer to use the keyboard whenever possible, I tried to install without using a mouse. It turned out that even though the focus on each screen was on the Next button, hitting Enter did not activate the control.

I opened a bug report. I even quoted Microsoft’s forms controls documentation:

“On any Windows Form, you can designate a Button control to be the accept button, also known as the default button. Whenever the user presses the ENTER key, the default button is clicked regardless of which other control on the form has the focus.”

It turned out that I am not the only one who can quote Microsoft. The developer dealing with my bug report responded in kind:

“The spacebar activates the control with input focus, whereas the Enter key activates the default button.”

If you think about it, you can see that the developer was right. On the other hand, I argued, the user expects Enter to work because this is how most other installers work. I was advocating that our program be consistent with the market, despite this implementation being technically incorrect. 

Eventually I convinced the development manager that consistency is important enough to justify the fix and that it trumps the Microsoft documentation.

It is easy to agree that consistency is important when discussing the user interface or the way in which a product implements actions that are closely related (for example, having the program use the same file-explorer interface when using Save As and Open). But I think there are other parts of a software product where being consistent is just as important, even though they are not as visible as the user interface. Additionally, I see consistency as one of the product aspects that we, as testers, have a responsibility to check.

Here are some areas where consistency is important but is frequently overlooked.


System interfaces—especially those that are exposed externally, such as the system’s published APIs—are a fertile area for consistency bugs.

If the application offers a rich API, it is almost guaranteed that the code for these APIs was written by a number of developers. Each developer has a somewhat different style and a different way of thinking about what is convenient, self-evident, and easy to use. This impacts the function names, the input and output variable names, and the order in which they appear in the function call. The result is that the consistency of the API package is degraded.

On top of this you will find simple mistakes that are the result of the developers knowing their code so well that they don’t notice when some small detail can mislead a person who encounters the API for the first time.

User Comments

Michael Bolton's picture

I'd like to emphasize a point you're making that some readers might miss if they read too quickly.

Consistency is important, for sure, but consistency is not a property; it's a relationship. For instance, in your first example, your product was inconsistent with comparable products, but it was consistent with the Microsoft forms documentation (a published and relevant standard).  In your "APIs" example, you're referring to the principle that if the API is inconsistent with itself, that inconsistency within the product is a problem that could reasonably lead to worse problems later on.  That's true, even though each individual call might have been consistent with the programmers' intended purpose.

Any suggestion of "no problem" in the product is based on the idea that an observation consistent with something desirable; any suggestion of a problem or a bug is based on the observation of an inconsistency with something else that is desirable. Trouble is, both consistencies and inconsistencies are in play at the same time. So rather than simply citing "(in)consistency", I'd like to emphasize the more powerful point you're making here: it's worthwhile to point out (in)consistency with what—and then, with stakeholders, decide which particular consistency is more important.

The FEW HICCUPPS heuristics is a set of desirable things with which the product may be consistent or inconsistent. I can't guarantee that it's a comprehensive list; it can't be.  I do believe that every tester should either use it or (better) develop a list like it, for at least two reasons:  first, to develop rich models of how we might recognize problems; too many testers focus on consistency with claims made in specifications or requirements documents. Second, to help remind testers when it might be okay to accept one kind of inconsistency when consistency with something else is more important to people who matter.


---Michael B.

July 18, 2017 - 12:46pm
Michael Stahl's picture

Michael - 

Some of the inconsistencies you point to (like inconsistency with expectations) are coming naturally to us as testers. Some others - and this was the main point I wanted to make - are things we may not always remember to consider. 

Thanks for the comments; as usual - they are intertesting and useful. 




July 19, 2017 - 1:54am
Michael Tomara's picture

I would add that the most convenient way to ensure a unified style of tickets and documentation is creating them yourself. According to my personal experience, in a team everyone follows their own style. Even if it has been agreed on beforehand, people tend to interpret the requirements in different ways. Of course a team lead can review each and every piece of content, but again, there are other priorities...

July 25, 2017 - 1:29am

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.