Let’s say you, a tester, are presented with a scenario and asked to design the test cases required. In this case, the customer describes the process to log into their newly developed application as follows:
- User enters their user ID
- User then enters their password
- If the user enters the incorrect password three times, the account is locked
Drawing on your knowledge of specification-based testing techniques, what do you think is the best way to approach test case design for this task?
Consider using decision tables in your testing. Decision tables allow you to create a graphical representation of the possible interactions and provide the information you need to develop an effective, efficient set of test cases.
Designing a Decision Table
Decision tables are used to test the interactions between combinations of conditions. They provide a clear method to verify testing of all pertinent combinations to ensure that all possible conditions, relationships, and constraints are handled by the software under test.
Continuing with the user login example above, to help add clarity to the requirement, you’d take the information provided in the use case and display it visually:
For the situation under test, there are four potential conditions:
- ID is valid—yes or no?
- Password is valid—yes or no?
- There have been three invalid password attempts—yes or no?
- Access to system is granted—yes or no?
To test each of those interactions, the full decision table contains sixteen columns:
But designing these sixteen test cases is only the first step. An additional layer of analytical review is needed to arrive at what is known as a collapsed decision table.
Refining Test Cases
When trying to test every possible combination, decision tables can become very large. The method of intelligently reducing the number of combinations from every possibility to only those considered "interesting" is called collapsed decision table testing.
In this technique, the combinations are pared down to those that will produce differing outputs by removing sets of conditions that are not relevant for the outcome. We remove redundant tests or tests in which the combination of conditions is not possible.
By adding this additional layer of analysis to the test plan design, the tester can provide more effective and efficient testing.
Looking at the full sixteen combinations we identified, we can now cut those down by eliminating redundant conditions and impossible conditions.
The first thing we notice is that there are multiple conditions where the ID is shown as not valid. We’d expect that all of those would be treated the same way, because the system will always handle an invalid ID entry the same, regardless of the password entered or how many times the invalid ID was entered.
Let’s highlight those conditions on our decision table:
Next, we notice that there is an impossible condition that can be eliminated: When the password is correct, there is no possibility that the third password attempt would be invalid. We can highlight that column in blue on the decision table:
Here’s another redundant condition: When the password is valid and it is not the third attempt, only one test case is needed to cover that condition. Let’s change that scenario on the decision table to green:
Lastly, when the user enters their password correctly on the third attempt, only one test case is needed in that scenario as well. This final update to the decision table is now in purple:
As a result, we can now see that there are only five test cases needed in the collapsed decision table:
This technique also produces a clean table that can be shared with management and stakeholders to help explain why the test scenarios were selected and how the number of test cases was reduced to make testing more effective and efficient.
Here is a sample output:
In the example used here, we were able to cut the number of test cases from sixteen down to just five—a reduction of nearly 70 percent! This is also possible in real-life situations.
Using Decision Tables Practically
Earlier this year, I was presented with a complex scenario where ten conditions would interact with one another. My challenge was to intelligently cull those one hundred test cases to a more reasonable number.
The first thing I did was create the full decision table, just like we went through in the example. After that, based on the requirements, we were able to eliminate half of the test case combinations right away because they dealt with a user already logged into the system, and we didn’t have to consider the situations where the answer to that question was no. That was a 50 percent reduction right away, with very little analysis involved.
After working with development and business partners to define requirements and further think about what we could get rid of, I was able to create a test plan for the application that was only twenty-two test cases—a reduction of 78 percent.
Decision tables are a powerful specification-based test design technique that can be used for a variety of scenarios. The graphical representation is also a great benefit for working with stakeholders and nontechnical members of the project team because it provides an illustrative example that is easily understood by everyone.
Taking the next step and applying the concepts described to reduce the possible conditions and create a collapsed decision table make it easy to further show management how effective and efficient this test technique can be.
Using decision tables is not always easy, but in my experience, the results reaped from the exercise are always worth the effort. Try this technique to make your test design process more efficient, effective, and understandable.
It's a great technique and gives great value but what's confuses me is the outcomes of TC3 and TC4.
TC3: How can someone get access to a system when password is not valid?
TC4: How can someone not get access to a system when id is valid, password is valid and have not made the third attempt?
TC3: the user has entered their password invalid 3 times in this scenario; this is testing to ensure that the account is locked after 3 unsuccessful password attempts
TC4: in this case, the 4th condition (access to system yes/no) is set to No, so the user has a valid ID and valid PW, but does not have access to get into the system
I must say that I also would expect a N here, that you don't have access to the system. Now you are indicating a Y
Tests also need to be non-ambiguous. I would recommend to split it up in 2 parts, conditions and actions.
You actually mean the Action of the system is returning an specific Error message that you are locked out.
I have read it again, Access to the system is actually an action, not a condition
I think you are misreading what "Access to system?" means; it is a condition not an action. In TC4, the person is entering their valid userid and password, but in the case of that being done after their 3rd failed login attempt they do not have access to the system. So even though I entered a valid user/password combination becuase I don't have access to the system I should see the account is locked message.
Though, I would argue that for TC3 I should also see an error message in addition to the account being locked.
Access to the system is unclear and I still don't know what is meant with it, that the user can do actions after login?
then it is still a response action
TC5 is incorrect, if I first type 3x the password wrong and then try a 4th time (third option don't care) with a correct one
I get access to the system according to the test, while I would expect to be locked out
If you use state transition tesing, could be more applicable. then you know if you have covered all the paths. Since the locked bit, would have been set
I agree, that is why I say "access to system" is a condition. When I hit the 3rd invalid attempt, access to system flag should become 'N'.
The article makes good points about decision tables and cutting down the number of cases, but the example leaves too much room for interpretation to ensure we have all the scenarios covered.
This is an old post, but I thought I'd comment in case any new readers stumble upon this.
My understanding was that the 'Access to system?' condition could be expressed as 'Is the authenticated user allowed access to the system?'; i.e. if the user ID is known and the password matches that of that user, are they allowed to acces the system. If that user's account was locked I imagine that condition wouldn't hold, otherwise it probably would hold (unless of course the user was in the database but didn't have sufficient privileges to access the system).
The second point that I wanted to make was that it isnn't always necessary to write out all combinations of conditions and then refine/prune. In this example we essentially have a logical AND between the conditions 'ID valid', 'Password valid' and 'Access to system' (assuming my interpretation of this last condition is correct). Using MC/DC coverage criteria we can derive variants of the test cases TC1, TC2, TC4 and TC5. TC3 is the special case where the account is locked.
The third point is that although decision tables are definitely useful, this example system would be modelled well by a simple state machine. You could then use simple state transition analysis techniques to derives tests such as the user being able to login sucessfully after the first or second failed login attempts.