Mutating Automated Tests

Summary:

Most automated tests are used as regression tests - doing the same exercises each time the test is run. This paper and talk describe a powerful type of automated test - one that does something different each time it runs.

These tests can augment traditional manual and automated regression tests to expose unexpected software under test (SUT) behaviors. The paper goes into the organizational issues and typical organizational evolution that are precursors for these tests and presents the idea of mutating tests. The approach doesn't apply to all situations of automated tests, but the author presents some pros and cons for mutating automated tests and provides several examples based on experience.

About the author

Douglas Hoffman

Douglas Hoffman is an independent consultant with Software Quality Methods, LLC. He has been in the software engineering and quality assurance fields for over 25 years and now is a management consultant in strategic fields for over 25 years and now is a management consultant in strategic and tactical planning for software quality. He is section chairman for the Santa Clara Valley Section of the American Society for Quality (ASQ) and is past chairman of the Silicon Valley Software Quality Association (SSQA), a 750-member task group of the ASQ. He has been a speaker at dozens of software quality conferences including STAR and has been chairman for several international conferences on software quality. He is also a member of the ACM and IEEE. Doug is certified by ASQ in Software Quality Engineering and has been a registered ISO 9000 Lead Auditor. He has earned an M.B.A. as well as an M.S. in electrical engineering and B.A. in computer science.

Although his current focus is in software test automation, his experience includes extensive consulting, teaching, managing, and engineering in the computer and software industries. He has 15 years' experience in creating and transforming software quality and development groups, and 20 years of management experience. His work in corporate, quality assurance, development, manufacturing, and support organizations makes him very well-versed in technical and managerial issues in the computer industry.