Exploratory Testing in a Regulated Environment


Exploratory testing can help you make the best use of your team's creativity, adapt to changes, and provide visibility into the decision making process. Don't miss out on these benefits because of an adversarial stance with your auditor.

Not too long ago, I worked in finance. My company made fraud-detection software for banks. It was complex, had a direct effect on people's lives, and was carefully regulated. We had multiple audits every year, which generally fell into two categories: risk assessment and regulatory compliance. I am not a lawyer and your industry's regulations may vary drastically from mine, so for now, let's focus on risk-based audits. Most of my experience with risk assessments has been as part of the Service Organization Controls (SOC) audit.

These audits involved a series of interviews and document reviews designed to answer the questions:

  1. “Do you do what you say you do?”
  2. “Has your company taken the appropriate actions to reduce your customers' exposure to reasonable risks?”

Depending on your business, this may include a review of finances, physical security, network and computer access controls, policies, and other documentation. Some auditors may be interested in shadowing you while you work and asking you questions about what you're doing.

In our case, after the auditors collected the necessary information, they evaluated our practices against some standards and wrote a report with their findings and recommendations.

Three Lessons I Learned
1. The auditor is looking out for your customers.
If you think of software as a formalized business process, then auditors are simply another type of tester. When I work with programmers and business people, I want my work to help them be better at their jobs. In the same way, the auditor wants to help you serve your customers better. The policies or regulations that the auditor uses for guidance were probably created because of situations that, while profitable for the company, may present an undue risk to your customers, investors, or the public. Try to keep that in mind as the auditor makes his recommendations.

2. Auditors don't dictate your processes. Think of the auditor as a coach. He will push you and maybe even challenge you, not because he wants to make your job more difficult, but because he wants you to grow. Take his words as advice. You probably know your company and its culture better than your auditor. If something doesn't sound right, ask questions to find out why the auditor has suggested you change your process. You may find that the advice is a lesson another company learned the hard way.

As you're talking to your auditor, you may discover that his background is in an area other than software—perhaps accounting. He may be unfamiliar with the nuances of various techniques. Our company's auditors wanted me to explain the testing phase of our agile project. I could have argued “We don't have a testing phase,” but the auditor would have heard that as “We don't test!” Instead, I explained how we are always doing some kind of testing. Our auditors could see the benefit when I explained specifically how we worked.

3. Know how you'll be measured. There's nothing like walking into a university exam unprepared, and the same goes for audits. Find out what laws and industry practices apply to your business and discuss them with your auditor. Your auditor may even have a self-assessment to help you identify areas of improvement before the audit.

Why Auditors Love Exploratory Testing
Auditors explore, too!
In the discovery phase of the audit, auditors are reading, listening, thinking, and asking questions about the information they encounter. They adapt to what they've learned. Sure, they may start with a checklist, but they're hardly confined to a plan. They analyze what they've learned and share it to help their customer make better decisions. If you re-read the previous four sentences with “testers” substituted for “auditors,” it sounds a lot like exploratory testing. Don't you think your auditor will support you when he realizes you work the same way he does?

Exploratory test documents tell a story.
Imagine you're an auditor visiting a company for the first time. You talk to a tester who methodically follows test scripts and reports which steps passed or failed. When you review the test results, how do you determine if the test was appropriate? How do you know how the test impacts the business? You would have a very limited understanding of the value of the test without also reviewing the system architecture, test design, and the decisions that were made after the test was completed. That's a lot of work, and you could only guess why this person did these things.

Now, imagine the same situation in which the tester's notes explain his thought process, what he wanted to test and why, what he didn't test and why, what he reported, and what decisions were made from the reports. This style of documentation gives a more complete view of the test. If I were the auditor, this is the kind of information I'd want. If the Sarbanes-Oxley Act applies to your company, your auditors will appreciate test documents that discuss how decisions are being made.

But my auditor wants extensive documentation!
Then give it to him! Following test scripts isn't the only way to document your testers' activities. Would he accept videos of the testers as they work, computer logs, screen captures, saved outputs, or some other evidence that the test was performed? An auditor asking for documents to prove that the test was executed may be more interested in verifying that you're performing the actions in the way that you say you do rather than in how effective those actions are. Our test policy stated that we would use a variety of techniques to evaluate software and provided an open-ended list of example techniques. As a general rule, I want my level of rigor to match my risk. If the project could result in people getting hurt, legal action, etc., I'm going to create solid documentation that can hold up in court.

That much documentation would be overkill for some projects, however. Remember the words “appropriate” and “reasonable” when you're thinking about your processes and risks. A short document explaining what you plan to test (charters) and how you'll evaluate it (oracles) along with a document summarizing your findings may be sufficient for less risky projects. My “test plan” and “test result” documents were usually less than ten pages combined for projects lasting two or three months. I also had an extensive archive of notes, videos, system logs, test data, and system outputs in case anyone had questions about the test results, but I never had an auditor ask for the details. If your auditor wants more, find out what he's trying to evaluate and provide information to answer his inquiry.

Making the Transition
Testers on your team are probably already exploring to some extent, so formally acknowledge it by mentioning exploratory testing in your procedure documents. To prepare for your next audit, include documents from exploratory test activities in your regular test documentation. If your auditor likes to shadow you, encourage testers to talk through their thoughts with the auditor. If some members of your team are uncomfortable talking about exploratory testing, consider using in-house training to help them be more confident in the skills they're using.

Exploratory testing can help you make the best use of your team's creativity, adapt to changes, and provide visibility into the decision making process. Don't miss out on these benefits because of an adversarial stance with your auditor. Reach out to him as a trusted partner; it just might make your audits a pleasurable experience!

Sarbanes-Oxley Risk Assessment Guide

SOC Audit Toolkit

User Comments

Karen Johnson's picture
Karen Johnson

Very readable article with no fluff and great tips.

I especially like the section on auditors and thinking of auditors as coaches. While my own experiences with being audited varied greatly from the author's experience, I could tell his writing reflected his personal experience and it helped to illustrate what another situation might look like.

Certainly a big takeaway from the article is documenting your thinking and decision process, a good tip made even better by using the words practical and reasonable. Yahoo! Documentation doesn't have to mean unwieldy.

I also liked Josh's realization and sharing his thought process when he says, if I answered a question one way, I knew it would be interpreted the wrong way. Instead I explained our process. Fantastic. I'm not sure I would have made that realization, or at least made that realization in the immediate on the spot timeframe! It raises the point that how you say something can make an impact on what gets heard and how from there a conversation can go downhill. Clearly the author saw and used the opportunity to clarify in a positive way.

Glad I stopped to read the article. Thanks to the author for sharing this experience. A great experience report.

September 24, 2013 - 8:04pm
James Christie's picture
James Christie

I can identify with this. I've worked as an IT auditor as well as a tester. I can back you up about what auditors expect. You're exactly right about auditing being an exploratory process.

I did wince at the two questions you mentioned, however.

1 “Do you do what you say you do?”

2 “Has your company taken the appropriate actions to reduce your customers' exposure to reasonable risks?”

I would never ask questions that allowed a simple yes/no answer unless I had the follow up ready. That would usually be "ok, show me" if the answer was yes. I would typically only ask that sort of question if I already knew the answer and was testing the honesty of the auditee. The first question would usually be replaced by separate questions "can you show me what you're supposed to do?", and "can you show me the evidence of what you actually do?".

The second question about reasonable risks would be replaced by a whole series of questions and analyses to work out what "appropriate", "reasonable" and "risk" meant in that context.

I had a fair bit of development experience before I moved into audit, so I was already well attuned to the problems of the people I was auditing. The company I worked for only hired IT auditors who had significant experience in real IT and then trained them in audit. That seemed entirely right and natural to me, but I know that it's far from being the norm. Pity.

This is a subject that fascinates me, which is why I'm scheduled to give a tutorial at EuroSTAR in November about testers working with auditors. It's an important topic and testers really should be proactive about speaking to auditors.

September 16, 2013 - 2:57pm
Michael Larsen's picture
Michael Larsen

YES!!! We need to have more people see this! It drives me crazy when I hear people tut-tut my suggestions about ET and say "oh, that would never fly here!" Now I have something I can point to and say "oh yeah?"

September 20, 2013 - 5:44pm
Timothy Western's picture
Timothy Western

This is an excellent article. So many think that having all the steps 'documented' covers it, but a test plan only shows one way the software may be tested, it can never document the implicitly hundred or so tests a second that the human brain does with out us realizing it. Anyone who wants to know more about this topic should definitely check out Griffin Jones talk at AST on What is Good Evidence at this years Conference for Association of Software Testing in Madison, Wisconsin. http://www.youtube.com/watch?v=i8he7Rejn5s&feature=c4-overview-vl&list=P...

September 20, 2013 - 9:09pm
srinivas kadiyala's picture
srinivas kadiyala

Thanks Josh Gibs, I have not read any posts with respect to auditors with testing.

In my prev company, there were people form organisation do as auditing. I was thinking at that time what will they work. Anyway it doesnt change anything.

Now,I think, if am right - auditing is done to help the clients.

can u correct me if i am wrong.

And Do if we write the notes, logs, and keep all testing documents - do they really ask testers?


Srinivas Kadiyala


September 21, 2013 - 4:03am
John McConda's picture
John McConda

Love the idea of auditors being explorers too. I'll echo what Michael said. I've heard testers getting excited about exploratory approaches but despairing because they think these don't apply to their regulated context. Hopefully, those ideas are slowly changing.

September 21, 2013 - 6:55pm

About the author

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.