As testers, we usually discuss how to approach testing when an application is created and maintained by a team within our own companies, but we rarely talk about what happens when changing from one outsourcing vendor to another.
In the world of IT outsourcing, it is not uncommon for a company to have its applications and infrastructure developed or maintained by others. As vendors compete for this business there are contracts involved setting up governance, service levels on response times, committees on release content, etc. A common part of the contracts is testing the transition activity as a whole, both for infrastructure and applications.
What the customer wants is the ability to document the service and solution they need so that they can outsource part of their process, usually for a better price or better strategic fit, without disrupting the business.
Recently I was a test manager on an application transition project with this challenge. The application had been maintained by another company since the ’90s, and we were to maintain and develop the application going forward. But before we could do that, the contract identified a “transition trial” we had to pass. My task was to make the acceptance criteria of the transition trial testable and clearly communicated.
What should that entail?
Testing during Application Transition Trials
There are some special terms in this kind of testing context. The contract defines it as a trial—not a test or a check.
A trial has many similarities to testing and quality assurance. We are given some high-level acceptance criteria and asked to confirm and document that these things have in fact taken place. Some of the specific acceptance criteria could be implementing some simple changes to the system, updating the documentation, and establishing recurring collaboration meetings.
Working with traditional software development acceptance criteria, we expect that the acceptance criteria would be about the application’s functionality and business-supporting activities. But the acceptance criteria here are regarding the item under test, so we have to frame our trial test cases accordingly.
One of the contractual requirements was the update of the system documentation. This was captured in the following trial test case: “Is the system description document updated?” We gave no further steps and no expected or actual results. It was up to the tester to evaluate if the document was indeed updated and for the customer to approve that we provided a document update.
Notice that we used open questions to allow for testing to be an activity and a performance, and the testing activity itself can be supported by guidelines and checklists rather than detailed scripts. It was very much a matter of “everything equal” and “nothing else bad has happened (that we know of).”
The acceptance criteria did not elaborate on whether the documents were updated or any other detail. What we could do was baseline the original version and then the updated version and look at the changes, at least in the authors section. Again, the testing preparations are an exercise in taking unclear requirements and rewriting them into testable activities.
A structured reading of the contract revealed sixty-five activities or documents, and with no need for additional test management tools, I set up each one as an open question in a line in a spreadsheet. Then I created four columns in the style of a kanban board: To Do, In Progress, Done, and Documented. The context and tradition of the company is that we have to provide evidence and documentation for each test activity.
Even with the traditional approaches to test analysis and test case management, many team members found it challenging that a test case could cover anything else but functionality coded into the application. Even the activity of doing a simple change to the system was never a test of the change itself—it could have been about painting the application pink, for all that matters. The test trial is more about being able to deliver a change than the change itself.
Expect to See More Transition Trials
Testing trials for outsourcing projects are becoming more common, not only with regard to applications, but also for IT infrastructure, IT support, and many other areas. There is a clear trend that Linux, Windows, Oracle, and other technicians are now required to do at least some testing of their configurations and customizations before releasing them for use.
These technicians generally look at testing as a confirmation activity, and they tend to use the V-model of software development, where each development stage has a corresponding testing phase. The good news is that testing is a required activity, and we can start having the discussion with these technicians about how to test, working toward testing as an activity where we learn about both the applications and the infrastructure.
Transition trials with acceptance criteria will become more and more frequent. As testing professionals, we need to be aware of the differences between testing an application and working with an application under a transition trial so that we can properly deliver a seamless conversion.