The testing of software is critical to the attainment of software quality. Reliable, meaningful
test results can only be obtained by a test facility operating within a properly structured
management system, using validated tests run by staff who understand the limitations of the
tests. Test laboratory competency can be confirmed through independent, third party
accreditation of the laboratory against the requirements of ISO/IEC 17025 General
requirements for the competence of testing and calibration laboratories. Such accreditation
supports software quality life cycle initiatives.
The development of software which meets the users’ expectations without exposing the developer, supplier or user to unacceptable commercial or legal risk is generally a substantial task. It requires significant planning, organizing, monitoring and control activities to ensure that quality software is delivered on time. Testing of the software is a critical element of the process used by management to monitor and control the quality of developed software.
How then can management be sure that the quality of the testing is adequate? Also how can the user of the software be confident that the testing of the software has been effective? This paper proposes third party testing accreditation as the means for developing heightened confidence in testing for developers, suppliers and users.
2. Software Risk
The risks associated with software are varied and range from the trivial to the critical. The consequences of flawed software range from inconvenience or irritation to the user, eg computer games, to major property damage and injury or death to persons who are not even direct users of the software, eg software controlling trains or aircraft. Even the impact of the trivial example may not be acceptable to the developer or supplier due to the potential for loss of image, good name, etc. The impact of the serious cases is usually acceptable to nobody.
Fortunately software has behaved quite well and the serious situations are relatively few. Presumably this can be attributed to the efforts of developers to manage the development process in a “quality” environment and to ensure an appropriate level of testing. However software is becoming even more pervasive in our lives. It is being applied in situations which were apparently satisfactorily handled in other ways in the recent past, eg. engine management systems in automobiles. Software is being used to create market needs which barely existed even five years ago, eg. DVD, WAP, and its development is being driven by market forces which are continually driving down the development and life cycle times. Software lifecycles pressures are increasing. New product releases are associated with market pressure for reduced development times and resulting pressure on the testing. Testing is often seen as the reason for software release delays—“ there were no problems until the testers got hold of it!”. An admission of poor management!
In this environment the need for “quality” testing becomes even more critical. We all hope that problems in the software development will be picked up by the testing, but if the testing is not adequate, or is not effective, what are the implications for software quality? Software quality cannot be guaranteed without effective testing. Poor testing will not prevent poor quality software being released.
Testing is usually the last chance that software developers have to discover software glitches before the users find them. Without effective testing, software developers would have a hard time demonstrating that they had adequately addressed due diligence issues. Effective testing also provides increased assurance for the suppliers and the users. They get to sleep at nights, as well as the developers. Finally the image and status of the testing community is enhanced by quality testing and the perceived value of testing is increased.
3. What constitutes a quality test result?
Some developers on a tight deadline might consider that if testing shows no defects in the software then this is a good result. Not only is this an unlikely scenario but it is most likely indicative of inadequate testing. It is suggested that quality testing has, at least, the following major characteristics:
- Capability. It must be capable, ie able to properly exercise the