In this article, the author explains how to test when you're under severe time constraints and you need to immediately provide an assessment of the system's quality, and when there are no written requirements to test against.
The term System Functional Testing speaks for itself. The "system" means that the subject of testing in the whole software solution. "Functional" implies that we test behavior, visible to the end users. In some cases, system delivery to testing can be broken down into several stages. But even if a part of system is delivered to testing we still can approach it using system testing model. For the sake of clarity, we will not give another name to testing a part of the system.
System Functional Testing is verification of the system against behavioral and non‐behavioral product requirements. Behavioral (functional) requirements describe how system reacts to the user input or environment change. Non‐behavioral (non‐functional) requirements usually represent some property of the system, in contrary to behavioral requirements, which, by analogy with OOP, represent methods or functions of a system. Examples of non‐behavioral requirements:
- System must run on Windows 2000 (.NET 2.0).
- System must comply with ISOXXXXX security standard.
- System must run unattended 24x7 with mean time to failure no smaller than 1000 hours.
Know Your Mission
As any other process test design works best when approach on several level of abstraction. System test design works best when approached top‐down. The topmost level is defining the mission. Just ask yourself what the product is supposed to do for an end user. What services it should provide. What performance and system requirements are? Make sure you understand the requirements and your mission before defining a test strategy.
Do not hesitate to ask questions if something is not clear. Asking them at earlier stages is much better than waiting until it's too late.
Test Important Things First
Put the most important things first. Define what the users are supposed to do with the system first of all, find the most probably paths through the system functionality to generate your first tests. When you are out of wits try Help system.
Aside from the sequence tests contain associated data. As far as our goal is to check the system in conditions close to how it will work in user’s hands, we need to select typical data sets. Typical data is what users likely will submit to the system.
But be careful not to fall a victim of faulty assumptions on your side. For example, you may define a tiny database for a typical data whereas users may have huge databases. If you do not know for sure, just ask.
Usually critical path tests cover 80% of functionality with minimal number of tests.
Extend Your Testing
Testing of critical path allows quickly removing serious obstacles for a more in‐depth testing. But it's weak against issues that become visible at boundary conditions or erroneous cases. Extended testing is there to cover this potential gap in coverage.
Extended testing focuses on extreme values and values beyond the extreme (erroneous cases). It also requires thinking in terms of path permutation (identifying and covering multiple logical flows through the system).
Extend Your Testing Even More
There is something wrong with human brain. It fails to create complex models. The more complex the system, the higher is probability that you will learn something new about the system at test execution. This simply means that you may need to add tests once you tried the system out on the real system. If you noted that there is an interesting test that you would like to try but didn't have written, just do the test and put a quick note down somewhere to get back to it and transform in a test case later.
The whole idea of exploratory testing is grounded on inability to model system behavior precisely enough, so as additional test design may be required during test execution.
Examples below demonstrate top‐down approach to functional testing.
Pretend we are working on a web‐based system that allows users save and share their notes to a group of other users subscribed to the system. System requirements state that an application should:
- Have a login system that allows users to sign up and sign in using login and password (user name contains only alphabetical symbols and is X to Y characters long; password can contain any symbols but spaces and is N to M characters long).
- Have means to change user password (old password shall be entered to confirm password change).
- Have means to accept user notes and save it into database (a user can allow or disallow making his or her notes visible to other users).
- Have means to allow users seeing notes of other users.
- System supposed to work with 1000 users.
- Each user may have up to 100 notes.
- UI description.
Let's put UI description aside and focus on main use cases for this system (what users will likely do right from the start):
- Subscribing for the service
- Logging in with correct credentials
- Submitting private notes to the system / viewing private notes
- Submitting public notes to the system / viewing public notes (including notes of other users)
- Changing private/public permission for existing notes
This is unlikely that users will change the password right after subscribing. Anyways there will be nothing wrong to include this requirement into critical path scope.
- Changing password
So, now we have use cases and all we need to do is converting them into test cases. Remember that on the data side we try picking values that users are likely to submit. User names and password would be typical combinations of symbols allowed by the application. Data submitted into notes can be a tricky thing though. Try to find out what types of notes a target user group is likely you submit and use this knowledge to generate notes content for testing.
Once we have finished test design for Critical path we can advance to the extended testing.
As you probably noted, we did not verify whether our system works correctly in respect to specified limitations on user name and password. We also have no idea on how it will work with a big number of notes available for viewing. So, let’s focus on boundary values:
- Subscribing with a name or password of minimal and maximal length.
- Adding and viewing a note of maximal size.
- Adding and viewing maximal number of notes to the system.
Now we have a much better testing coverage than if we had only critical path tests. However, it still has some weakness. We do not know how system will react if user tries to use incorrect symbols for user name or password or if its size bigger or smaller than specified.
- Subscribing with an empty name or password
- Subscribing with a name or password of minimal-1 and maximal+1 length
- Adding a note bigger that maximal size.
- Adding maximum+1 notes to the system
- Subscribing with invalid user name (containing nonalphabetic characters)
- Subscribing with invalid password (containing spaces)
- Viewing zero notes.
Note that the latest scenarios we included in our testing examine logic branches through the system. Whereas correct user name and password lead to successful completion of the operation, the variants we have chosen should lead to an error.
From the perspective of a visible user behavior, it does not matter what specific type of input have caused an error. If error is the same then all types of data that caused it form an equivalent class (a class of equivalent values). Just the same as all typical correct values form another class. This is always useful to distinguish values near boundaries into a separate equivalent class, as well as such values like 0, NULL, undefined, etc.
Now are testing strategy looks almost completed. “Almost” is here because it is built on requirements. Requirements are not always complete if it comes to possible user actions. Do not refrain yourself by the limits of product requirements only. Think beyond. Be creative.
This is a process that makes testing not so easy but also adds so much fun into it ☺
Here are some cases that will demonstrate "thinking beyond what is written" approach:
- Tags and even scripts in notes would be useful.
- Same user logs in twice from different systems and tries manipulating same data.
- Notes, user name and password in a different language (not western European).
- Turn down the database or lock data file to see what system will write to the end users. Make sure the message is informative.
So, this is it! Now I am pretty sure that we will not miss functional issues should they exist. However–to make sure that there is nothing you simply do not know yet–give your testing strategy for review to Dev representative. Based on the knowledge of code structure, they may probably pinpoint where to look for additional equivalent classes.
The Real World
In the reality you may not have enough time to do all above. Sometimes you are just asked to do things quickly and provide your assessment of system quality immediately. There are no written requirements and there is no time to collect and write them down to the paper. Welcome to the real world! If we (as a company) have failed to sell an adequate testing to the customer, you have to do all the best, everything possible and more, to provide the best testing service that the circumstances allow.
For example, if you have no requirements. Well, you need to ground your assumptions about the system on something. Ask project manager or, on his permission, build a direct communication channel to the customer to find out how system works. Even if you can assume how it should behave in the give case ask customer to confirm that your assumption is correct. Guess but verify!
Now, once you have knowledge how system should work you may have a hard choice. On one hand, you want to do things right the first time and to generate written test cases, at least for the critical path testing. On another hand, you were asked to provide test results this evening. Ok. No time to play dolls. Bring it on!
Start from what really matters. Do tests for things that a customer is going to see first. Proceed until you have time. Report problems and risk immediately. Write notes as you do your testing. It will help you cleaning the mess up, once you have time for this.
If you left something not tested (many items, big volumes of data, stress, load, etc.) include it in your report. Let project manager and customer know what the risk level is. Share information. Share your assessment. Share your concerns. You are here not to please people with positive test reports. You are here to guard customer’s interests. Why? Because there is no one else to do it ☺
The Regression Testing
In scope of system functional testing, regression testing is repeating selected functional tests in order to verify that functionality that worked correctly before the change still works correctly after the change. For example, developers fixed a defect. Aside from checking that the steps described in the defect report do not lead to a problem anymore, tester has to make sure that adjacent areas are not broken by the fix. The reasons why you need to test for it are quite different. System may have similar functionality in another module that developer overlooked. Or a core component was fixed and most of the functionality built upon it is under risk. To make sure no new issues are introduced, we do regression testing.
The same is for system functional testing. Imagine, you've got a new build and developers tell you that they have introduced a new functionality. Besides testing for new features, you have to do regression testing for the old functionality that could be affected. Where to start?
As usually in testing, we start from collecting information. Information helps taking on right decisions. Ask developers where they recommend focusing attention while doing regression analysis. Unlike you, they know code structure (or at least pretend to know it☺). Based on this knowledge, they may point you into the right direction in your search. Do not rely on guesses only. Way too often applications fail in the parts a tester would never imagine as being connected to a change.
If you are lucky and you have working automated tests, then your regression analysis will be as simple as it takes to decide on executing all the tests.
Testing still requires quite a bit of problem‐solving skill and creativity on your side. This is why it is still so much fun ☺
Do not stop thinking! Never!