The 11th Hour

Testing Before the Release Goes Live

Testers are often on the critical path for getting a software release out. They must plan carefully in order to minimize the critical path, while still doing a complete job of testing. This schedule pressure is taken to an extreme when a production server must be taken offline in order to deploy the software, and everyone is waiting for the final test results before the system can go live again. Karen Johnson describes her company's carefully planned and orchestrated method for doing a final check of an installed system. Her story is relevant to e-commerce companies as well as IT shops that are under pressure to keep systems updated while minimizing downtime.

Beyond industry knowledge and technical skills, a quality assurance person brings to the table a critical career skill that has nothing to do with the latest or coolest technology: the skill of planning. 

It's important to be organized in a profession that often senses other people (such as the rest of the technical and management team) pacing and waiting for QA to finish their job. Quality assurance needs to be clear with the rest of the team about what needs to be done and the risks we take when corners are cut because we are short on staff or short on time.

As if the development process didn't put enough time pressure on quality assurance, the release cycle turns up the heat even higher. During the development cycle, new functionality and features are created and tested. This typically involves weeks or months of build, test, and rebuild cycles. During the release cycle, the new code is ready to be delivered and the final CD is cut, or in my case, installed onto a Web site. Quality assurance is the final step before a release is placed in the hands of the customer.

Life on the Web
While the theory and promise of a 24/7 Web site are talked about, the reality for many company Web sites is that they need to be taken offline (at least occasionally) to install new releases or upgrade software or hardware. In my environment, we occasionally need to take our production site offline for an hour or two during the night to install a new release. A variety of people are involved with these releases and the pressure is high-everybody takes their turn working through their piece of the process, knowing that offline time means lost revenue and reduced customer service. In this race against the clock, quality assurance is designated the last position-the final checkpoint before the site is open to the public again.

For us, these middle-of-the-night activities begin during the day. We collect statistics on our site, learning what day of the week and what time of day our site has the least amount of customer activity. This is how we select what time to take the site offline. When the install time is selected and the new release is ready for install, we hold a release meeting. We map out a tight schedule of what needs to be done and who needs to complete each task.

When the magic hour approaches, our network staff begins by trying to close the site as gently as possible for customers-waiting for people to sign off and hoping more people aren't logging in at three o'clock in the morning. Once the customer count is as low as possible, we post a maintenance page stating that our site will be offline temporarily, and an email will be sent when the site is available again. I use a connection outside of our firewall to verify that this maintenance page is what is being shown to the "outside world." I enter my email address to receive notification when the site is live again. Then, the network staff makes any hardware or software upgrades as needed, followed by configuration changes.

Meanwhile, our database administrators (DBAs) begin making modifications to the production databases. Database tables may be added, changed, or deleted. Sometimes these database changes can only happen when the DBAs can gain exclusive access to a table. The only database changes that are made are those agreed upon in advance by development, quality assurance, and the DBAs. These database changes are well documented and executed according to the agreed-upon


About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.