Noel: Hi, Jon. I wanted to give you a chance to talk a little about your new book, Software Test Attacks to Break Mobile and Embedded Devices. What made you want to write it? Honestly, it seems like a great time for a book like this, given that embedded software is on everyone's minds these days, whether they know it or not. And please feel free to let the readers know where they can find your book!

 

Jon: Thank you, Noel. The book is out now and is available in both e-reader and hardcopy format from various sellers, like Amazon, Google, and the CRC Press website.

 

I think the time is ripe for a book specific to testing mobile and embedded software devices, given the growth of software in these. We are seeing “smart” systems everywhere: cars, planes, light switches, TVs—you name it. If it is electronic, it seems somebody wants to make it “smart” or smarter.

 

Likewise, the mobile and smartphone devices with apps are growing at high rates—much faster than traditional computing platforms. Such explosive growth is familiar in the high-tech world. We saw it first in computers, then personal computers, and then the so-called “dot com” web world growth. Each time it seemed with the rush to market with software, quality took a back seat. But then, not every company producing software in these explosions survived. Many of the companies that did survive balanced quality of software with cost to produce and time to market. Testers should provide the key information about the qualities of software.

 

There is no one answer about what a "quality key” is, and there is no “best practice” in testing, but for companies producing mobile and embedded software, they must consider using testing to help balance the quality-to-cost-to-schedule equation. Companies need to see test not as a tax, but as a resource to provide decision makers with useful data points with which to make decisions.

 

I wrote this book to capture aspects of the mental models I had developed over thirty years of testing mobile and embedded systems. I came to realize that I had a mental error taxonomy that guided my testing. I spent years searching public records to first create a written taxonomy. Then I published the data (and it is in the book), and it won "best paper" at a STARWEST conference in 2010. Next, I started thinking about James Whittaker’s work on attack-based testing and realized that my taxonomy and the mental models I had could lead me to doing attacks the way Whittaker described, but my attacks were different because embedded and mobile had unique sub-capabilities and bugs.

 

So, I started writing and creating attack patterns to address errors that the taxonomy indicated were common. The attacks expanded in the areas where mobile and embedded devices are now going, such as security, gaming, communication, performance, and controls. I did years of work on this while embedded and mobile software grew. The mobile and embedded software world is a moving target, but I believe the book serves as a starting point for testers to find patterns to apply in their testing, and then to expand into their own mental models. Finally, I hope the book can help traditional IT testers move into the mobile-embedded world.

 

 Noel: That's a really amazing amount of research. One of the many great topics that you cover in this book deals with "forcing unusual bug cases." Two questions for you: What makes certain bug cases "unusual," and secondly, what are some ways that these are "forced" out into the light?

 

Jon: So, for the first question: Embedded systems, now even mobile smartphones, are often controlling something, such as hardware, airplanes, cars, and even hearts, in the case of pacemakers. Control of these things is done in the “real” world, which is basically complex, if not infinitely complex, in the varieties of situations. Programmers try to anticipate these situations, but obviously they cannot address every situation and program for it. So, unusual bug cases can hide in at least two ways. First is the case where the developers just miss a likely scenario of control—say, for example, a case where the stability control system of a car must handle a slow skid in a turn on ice. If the logic to handle this case is not coded, the system has a case, maybe not overly common, which is not handled.

 

The second case is where the code contains the logic to handle a situation but some logic was not coded correctly, and the team never runs a test with exactly the right conditions to trigger the bug. It is estimated that in many embedded systems, upwards of 80 percent of the code is dealing with such “unusual” situations. This is where the bugs hide, waiting for just the right situation to happen before they are exposed. Therefore, many of the attacks in the book deal with patterns to force test into these types of “unusual” situations.

 

In terms of the second question, in the book there are numerous stories and sidebar examples of where more testing might have forced the bug to be found. I mentioned the stability control system of a car, which was a real case. Bugs like this can cause recalls, which are expensive and can drain company profits, and cause bad publicity, which is maybe worse. So, how much more “good” attack tests might have been justified is a question that projects must consider. In the situation where some segment of code has not been tested under the right conditions, I can cite the story from the book of the patriot missile system, which had been in use for years but when left on for long periods of time had an error factor which accumulated over time, with the result that the system failed and people died. Who wants to be the tester that worked on that missile just before that event?

 

Hopefully, the attacks in my book will help testers to see differently, to try new things (i.e., tests)—and in the end, companies will profit and software will be better.

 

Noel: I'm not just trying to state the obvious here, but how vital is quality software testing in relation to embedded devices? We've gone from software in our phones to our cars, medical devices, our entire homes. There's this great race to essentially embed everything. Do you feel like those organizations and corporations that are having these devices developed are taking the testing of these devices seriously enough?

 

Jon: First let me say that many organizations I have worked with or supported take testing of these kinds of devices that you mention very seriously. When there is large risk in the form of losses such as money, life, or other resources, much good testing gets done. Many companies operate under regulator rules or threat of legal actions. But what my study of bugs that made it into the field and usage showed is that there were patterns in the errors where we as software people can do a better job. Developers and testers both missed perhaps hard-to-find bugs. In other cases, maybe some industry areas where they were less than diligent in their testing, different bug patterns appeared in the taxonomy. In both of these situations from the taxonomy, my study indicated to me that improvements could be made by inclusion of risk-based attack test patterns. After I did a detailed public error taxonomy study looking at years of data across several embedded domains, I came to the idea that attack-based testing could offer an improvement of ideas that a tester could consider.

 

 Noel: I read all these stories online of testers having to prove their worth and trying to justify their need in the software world. It's horrifying to think that testers aren't being regarded so highly when it comes to the prevention of keeping something like pacemakers from being hacked into or planes being brought out of the sky. Who do you think will be ultimately credited for giving testers the respected position within an operation that they deserve?

 

Jon: I might suggest that testing, or any single development effort, should never be considered "priority one." The problem with calling something a “number one” priority is that each group tends to think their job focus is undervalued and should be viewed as more important or a number one priority. I teach and talk with business analysts, proposal teams, system engineers, software engineers, hardware engineers, programmers, testers, QA, CM, and others. Each feels their contribution is undervalued. Being an agile supporter, I would advocate that each of these disciplines is of equal importance and value. Each group needs to focus on providing value and quality to the overall product and project. Project personnel and even managers both need to look for balance—and this includes testing.

 

Now, to the implied problem of testing being called “second class,” of lower value, being paid reflective of that perspective, and being perceived as a non-value-added “tax” on development, which some managers or other stakeholders might wish to be rid of: We hear the cry “Testing is dead,” not just in embedded and mobile systems, but with other software too. Who will change these perspectives? I think some kinds of testers may have an endangered skill. If all you can do is write a detailed test procedure based on requirements and run that test over and over at the end of the development of the embedded system, you might want to rethink your job, because your days may be numbered.

 

However, in the embedded and mobile world, if you:

 

• have skill and knowledge in system, hardware, and software engineering
• can provide valuable quality information during the development
• can support the team in rapid development with, hopefully, a fast project closeout testing effort (measured in perhaps hours or days, not weeks or months) to tell stakeholders the device is ready to ship

 

… then, my answer would be we as testers control our own worth and respect. Of course, this is just my old, jaded opinion. In the cases of mobile and embedded software, I discovered that as a tester, the information that I provided to the whole team was valued and respected. In the book, I tried to capture the things (attacks) I did to get this respect.

 

Noel: You yourself ask a great question right in the first chapter of your book:

 

But why cannot we just use the testing common to other software and systems? In the 
next section, you will discover the differences. Testers must have additional attacks in their 
toolboxes with all of the differences and unique bug patterns. The tester may use attacks 
from the How to Break Software book series, but new attacks should also be considered.

 

Why can't we use previous testing mindsets, tools, and strategies? What about mobile, embedded systems provide a unique challenge that "common" or "traditional" testing methodologies just can't cover fully?

 

Jon: In part, the tools, techniques, strategies, and mindset of traditional IT testing continues to be applicable in the mobile and embedded software devices. Testers in the mobile and embedded space should not forget classic testing concepts. My book cites and cross-references to books and attacks by James Whittaker and others that were aimed at the PC/IT systems. I also discuss many test concepts and techniques that traditional testers will find familiar. But here again, the taxonomy showed error patterns which lead me to define attacks in areas that were different than the ones talked about by James Whittaker. A tester moving into testing mobile and embedded systems must be aware of these different error patterns and factors so they can start to update their mental models of where bugs hide.

 

For example, many mobile and embedded devices are very battery-dependent. How many IT software testers have ever thought about the ways to test battery performance from a software perspective? And yet we know that software can directly impact the battery usage on a mobile or embedded device, and users really care about battery life. The book lists and examines attacks on such areas of mobile and embedded devices, including the mobility itself, batteries, environments, network connections, resource limitations (CPU speed, memory number of apps, etc.), and device characteristics like screen size, user interface, hardware interfaces, etc. A tester moving from traditional IT/PC/large computer testing to the mobile and embedded space can quickly get ideas for improvements in their testing because of the different mobile and embedded contexts. Context matters.

 

 

Jon Hagar is a systems software engineer and tester supporting software product integrity, verification, and validation with a specialization in embedded software systems. For more than thirty years Jon has worked in software engineering, particularly testing/verification and validation. Embedded projects he has supported include the space shuttle, large rockets, spacecraft, and ground systems as well as testing new smart phones. Jon has managed and built embedded test labs, including supportive automation development. Jon publishes regularly with more than fifty presentations and papers in software testing, verification, validation, agile, product integrity and assessment, system engineering, and quality assurance. He teaches classes at the professional and college level.