How to handle errors in test cases during execution, in a regulated environment?

Share URL
 
Kajo Johns's picture
Kajo Johns asked on October 21, 2014 - 3:55am | Replies (2).

Hi all,

I have a question about how you all handle errors in test cases found during test execution, during a formal System Test phase, in a heavily regulated environment (Medical Device Software - ISO13485, IEC62304)?

My instinct was that I should capture errors found in test cases (where they don't match the requirements/design assumptions/have incorrect 'expected results' or test steps are incorrect) as a defect (type 'test case error'), assigned to me or the test case author, to be 'fixed/resolved' by correcting the test case, after the end of that test cycle. This would mean we log effort spent in correcting duff test cases against the defect raised for them, and can prioritise that defect according to available resources in the next sprint.

However, my software dev manager disagrees and thinks we should just correct test cases on the fly, during execution, as we find any errors with them, so the tests just pass within that test cycle. He says any effort 'fixing' test cases should be logged against our 'test execution' task for that feature. He fears an explosion of issues/defects in JIRA, and his argument is that if he were to find a software bug during unit test, he would just correct it on the fly, without raising a defect... 

I'm uneasy about this approach for a few reasons
1. I want some evidence of *why* a test case changed after the test execution activity started
2. I want to be able to see how much time is being spent correcting test cases - to trigger some process improvement in the review stages if this turns out to be a significant number of hours
3. If we end up executing a different set of tests than was planned at the start of the sprint (same tests, but with different 'expected results' because the tester changed them 'on the fly' during testing), wouldn't auditors have something to say about this?

Has anybody got any thoughts or experience on this? The key is that we are heavily regulated and need evidence of everything we do.

thanks for your help/suggestions

Tags: 
defect tracking
quality assurance
test execution
test management
test process

2 Answers

Dan Pautler's picture

Records of verification, which I will assume you are using these tests as a part of, are critical in any audit / inspection of design controls. In every design control audit and inspection I have been through, good verification records have always helped lead to positive outcomes.

If the testing you are running will become part of your formal record of verification, then you should have records showing why and how those test cases changed over time. Using a bug report against the test case is one way of doing this. The record should include why a change was needed to the test (in this case the bug), what change was made, who made the change, who reviewed and approved the change (the review/approval should be someone other than the person making the change). Test cases are no different than other documents in that the test cases must be reviewed and approved before they can be used and any changes to those documents must be subsequently reviewed and approved.

You also need a record that the updated test passes. You could have someone approve the necessary modifications as you are running the original test and then the subsequent review is to verify you documented what was done in the test case, or you could update and run the updated test case prior to completion of the verification. I am sure you can come up with other options as well to satisfy the regulations.

Kyle Ensley's picture

I'll try an disect this as best I can in this online format

1. I want some evidence of *why* a test case changed after the test execution activity started

This is a reasonable request and in my opinion a requirement. I would be more concerned if there was no evidence. I personally maintain a log in a confluence page that contains sections of testing steps with space for notes attached to them. Each time a deployment happens a new log is made to record these and keep track. What I do is if i see a test case either fail or have unexpected errors I will log this in a public test checkout and include as much data as possible.

2. I want to be able to see how much time is being spent correcting test cases - to trigger some process improvement in the review stages if this turns out to be a significant number of hours

You can do this with a either a checkout plan that stubs out each section of your testing suites. Or you can break these out into tickets and log hours to those tickets. I currently use Jira and will log time to deployment tickets and add notes to that log time of what specificly im doing. You can use this data to create tables and Charts to have a visual of what your time usage is looking like.

3. If we end up executing a different set of tests than was planned at the start of the sprint (same tests, but with different 'expected results' because the tester changed them 'on the fly' during testing), wouldn't auditors have something to say about this?

Sometimes you have to make changes on the fly, but this should never be regular practice. You have to create a baseline to compare against. If your test suites are automated this is easy to do. Run your baseline, make the changes run it again.

Best thing to remember is to test for what the user is expecting. I'll spend less time testing specific dev changes and spend more time making sure the client/user is getting what they want.

A followup question to this is are your devs only doing unit testing? or are they actually testing the physical application as well? The reason I say this is unit tests can be fudged to pass, making them unreliable. If unit tests are utlized correctly they are a fantastic tool. But nothing can replace the real deal completely.

I understand what your dev manager is saying. Fixing unit tests on the fly and limiting the number of tickets. This is why it is important to have seperate regions for Dev and QA. THis is also why branching and merging is important. I am generally ok with with devs fixing things themselves if they find the issue in their own environment and its not present in higher regions. The second it bleeds into a higher region of code, I expect a defect. 

Good Luck!

StickyMinds is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.