Testing in Internet Time

You're a quality assurance professional working for a dot-com and everything you do violates the best practices with which you're familiar. Requirements are minimal or even non-existent. The test cycle is so short there is no time to write a plan or develop a test strategy. Most of the testing is ad hoc. How do you ensure a high quality product and still maintain your sanity in this environment?

First, let's take a look at the problems. Much has been written about the trials and tribulations of web development. Everything from FAD (frantic application development) to "requirements, we don't need no stinking requirements". And most of it is true! After all, you are but a mouse click away from losing a user so the site has to be continually updated. The business needs must drive development if you want to stay in business and the users don’t know what they want but they are clear on what they don't want and don’t like!. There is no time for the traditional waterfall methodology's stages–plan, design, code, alpha test, release to a couple of customers for beta test, then finally general release–a cycle taking up to a year or more. There is not even time for the spiral development approach where prototyping becomes code, testing is on going and releases are frequent. There is just no time.

So how do you work in an environment where quality is clearly important because a bug can drive a customer to the competition, where requirements are often articulated only when the developers or testers ask critical questions, and where testing time is measured in days or even hours?

Here are some ways to cope.

First, accept the inevitable. Your dot-com is probably struggling to survive (and you do have a job where many do not), so focus on being flexible and pragmatic. I can hear the sighs of horror. I don’t mean be chaotic and disorganized. Just be flexible and pragmatic. Accept that the requirements are undocumented, inadequate and subject to constant change. Accept that you will begin testing before the code is anywhere near ready. Accept that your test time will be measured in days at best, perhaps just in hours. And find some ways to help yourself, your team and your company.

How can you do this? Use the time before and after a testing cycle wisely. This is your planning time, recovery time and the time to fix the things you did wrong the last time. Use it to plan as much as you can. Get your regression test series in order. Fix all the problems noted during the last test series. Make all the updates. Write the test cases you didn't have time to write during the last cycle. If your company owns automation software, automate regression tests in stable areas. Do anything that will make you more efficient when you begin the next round of testing.

As you write and review test plans and test cases, note the level of risk that not testing in that area or using that test case represents. Develop a scale for risk evaluation, something simple such as a one to five scale where one represents catastrophic risk and five represents almost no risk at all. Start by doing this for each area you test. As you become more confidant in the methodology, do the same thing for each test case. Review test plans, test cases and test scenarios with management, developers, users or user stand-ins and anyone else whose is involved in the development process. They may disagree with your assessment of risk. Let them help in evaluating the most risky and least risky areas. Then, during the next test cycle you will know which tests to perform first (those that focus on the most risky areas) and when all testing cannot be accomplished in the timeframe available, everyone will know, and, hopefully agree on, the level of risk.

Use other people to help with the testing. What, I hear you say, let non-testers test! This is a crazy idea but one that has merit. The immediate advantage is that there are more people testing so there are more chances of finding problems. Sometimes untrained testers find things that trained testers miss. These people will probably operate more like users so that's a plus. Don't give them the complex stuff that takes lots of set up time. Give them simple stuff, some of the regression tests that you've done a million times anyway and are marked as low risk areas. Assign one person to coordinate with the non-testers so that the whole test team is not diverted from testing. That person can help with problems, answer questions and report any bugs they find. Remember (and get your management’s agreement to this) that anything they find is a bonus. They are not replacements for real testers. They are foot soldiers beefing up the attack at a critical moment. Once the test team members have overcome their horror that other people are going to be doing their jobs (which they are not, they are doing only one small piece of the j job) they will realize several things. They will learn that the non-testers are finding bugs the real testers would have found anyway, they were just found sooner. They will discover that people have learned some things about testing and the development process in general that they didn’t already know; and that most people have new-found respect for testers and for what they do. I'd like to add a couple of cautions here. Developers are not the best substitute testers. They tend to become easily bored with testing, and, more important, they think about software from a different perspective so they don't find the problems that users will find. In addition, they've already done (or should have done) unit and integration test so in their minds there is nothing left to find. My second caution is that the test cases you give to the non-testers must be well written and complete. Fix them during the downtime and include the improvements suggested by your helpers so there are fewer questions with each test cycle.

Develop a "shorthand" test case format for areas that are changing too rapidly too late in the cycle. This just outlines the combinations of factors to use for testing. It may even be a matrix. These areas will be tested by experienced testers who can track what they did for record keeping purposes without taking a lot of time. These shorthand test cases will be turned into detailed test cases later, during a lull. This is also a time to allow your intuitive tester(s) do some serious ad hoc testing, again with documentation to follow. Some testers, and if you are lucky you have one or two on your team, can open a window and know there is a problem there. Use that skill in the crunch.

How can you overcome the problem of poorly defined requirements? Before, during and after the testing cycle, stay very close to the people who are close to the customers. These are usually the people who have day to day customer contact. In organizations where I've worked these people have usually been trainers and customer care representatives. They can help you to understand who the customers are, their jobs and their skill levels and how they actually use the software. This will provide you with the background information to make rational decisions when testing new functionality. Similarly, stay close to the developers. They can tell you why they coded what they coded and you can tell them about other areas of the software and other functionality that they might not have been aware of. None of this is a substitute for a well considered requirements document and a detailed specification. But it will help you to maintain your sanity during those stressful days of testing.

My last two recommendations are the post-mortem and root cause analysis. A post-mortem allows you to find out what can be improved and how. This is useful if everyone accepts the realities of web testing are not focusing on the way things used to be done when they were testing in other environments. Along with this, go through the defects for each release and find the root cause of each. You, and your management, will be surprised at how many defects were not coding errors but were caused by lack of carefully developed and well-documented requirements. And this is you best tool for improving the requirements definition process.

Remember that you cant test web-based software effectively if you are not working in internet time. So forget how it used to be done and adapt to the new ways and the new age.