How often are software testers getting lucky?
By that, of course, I mean how often is our software simply not getting used and abused enough to show whether or not we’ve tested it well?
Take industrial control systems testing, a subject on the radar due to security incidents like the Stuxnet worm. While the Stuxnet attack was sophisticated, researchers and security hobbyists, like the man who pinged the whole Internet, have demonstrated that significantly less-complex vulnerabilities abound. That these vulnerabilities have not been exploited to more detriment of the public speaks not to real security or to the quality of the testing done but rather to the disinterest and perhaps patience of potential attackers.
In December 2012, Chinese hackers infiltrating a decoy water control system for a US municipality were caught by a research project aimed at exposing hacking groups that intentionally seek out and compromise water plant systems. According to security researchers, none of the attacks exposed by the project displayed a high level of sophistication. Researcher Kyle Wilhoit concluded, “These attacks are happening and the engineers likely don’t know.”
Vulnerabilities like these could permit attacks resulting in power outages, environmental damage, and loss of life, yet hacking industrial systems remains quite easy for hackers versed in the workings of them. Of three presentations on control system vulnerabilities at the Black Hat computer security conference earlier this month, all required significantly fewer resources and skill than what was needed for the Stuxnet operation.
But security is tough business, multi-faceted and fed by a small pool of technical talent. Add to that the low incentive value for energy operators and manufacturers of control systems to beef up security and it’s not that surprising that massive vulnerabilities exist.
Even where there’s huge liability and money on the line, however, projects still fail in the wild related to non-security bugs that testing should have caught.
J.P. Morgan, one of the largest banking and financial services firms in the world, lost its rear end over an Excel-based value at risk (VaR) modeling tool designed to help the chief investment officer (CIO) understand the level of risk the company was exposed to, which would help the bank make decisions about what trades to make and when. After a poorly positioned trade resulted in losses that totaled into the billions of dollars between April and June 2012, an investigative committee found that the tool that gave such horrendously bad advice had been poorly designed, developed, and tested.
A J.P. Morgan report detailed how it happened: “Specifically, after subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR…It also remains unclear when this error was introduced into the calculation.”
Even a rookie tester with half her brain tied behind her back should have found such a grossly incorrect formula that caused the firm’s VaR to be misreported by a factor of 50 percent. But the investigative committee found that “inadequate resources were dedicated to the development of the model…the model review group required only limited back-testing of the new model, and it insufficiently analyzed the results that were submitted.”
The committee’s report indicated a complete lack of rigor in testing by all who touched the modeler. “Data were uploaded manually without sufficient quality control,” the report claims. “Spreadsheet-based calculations were conducted with insufficient controls and frequent formula and code changes were made. Inadequate information technology resources were devoted to the process. Contrary to the action plan contained in the model approval, the process was never automated.”
The massive software failures we see from time to time and the known vulnerabilities of our systems bid the question: How solid is the house? Has our software been battered by the elements enough to prove that it is as solid as we think it is? In other words, are we as good at testing as we think we are, or are we just getting lucky?
Many of us have probably earned rock-star status the first week or two on a new job by finding a truckload of bugs on the existing codebase that the project’s testers never found. There’s something to be said for fresh eyes, but if a new tester is bouncing in bugs like a toddler in a ball bit, most of those will be low-hanging fruit and the heuristics and skills used to find them nothing more than elementary. Lest we be tempted to elevate ourselves as the crowd cheers, it shouldn’t be missed that many folks have simply never seen a good tester in the flesh before, and much of our software has never been rigorously tested.
Competition makes for bigger, faster, and stronger testing. The challenges are how we test our mettle and grow in hopes of rising to the occasion. But what if there are few challengers out there in the trenches? We don’t want to deploy weak software—like that used in industrial control systems—that looks tough on the outside but breaks down the moment it is pushed. How can we become better testers when we have no mentors in the flesh or peers to test? Besides, who can chastise us across the cubicles for missing an obvious bug?
One suggestion is to be proactive building your team, whether you are in a position of leadership or not. Buy a couple of testers lunch, grab a conference room, and teach them something while they are munching away. Type up a handout with the exact steps and commands needed to do something, like search for a specific transaction in a log. Break learning down into bite-sized chunks. Your team members will be energized by their new powers of testing and your skills will grow by teaching something that was formerly just in your head.
Another suggestion is to actively talk about your software to people outside your immediate test team. Hopefully, you are talking regularly to the developers, but if not, now’s the time to start. Talk to developers, database administrators, security pros, performance experts, and the server team about your project. Each of these people plays a specific position on the team. They may not be able to tell you what the guy playing first should be doing, but they can tell you everything about what the short stop should be doing because that’s the position he plays. Talking to a variety of IT pros rather than limiting your conversation to your circle of testers or even developers will help you figure out how to truly exercise your software and make it stronger before it gets deployed.
Start a blog and write about what you’re testing, minus any details you can’t give out. Pose questions on Twitter. Get a conversation going about some real testing effort you’re engaged in. Don’t be afraid to look like a non-expert—someone out there probably is a better tester than you because they’ve been in the game longer or encountered more. You can leverage this person’s experience to exercise your skills.
As testers, some of our track record will be pure luck—for better or for worse. We should, however, strive to test well enough that users must be crafty to cripple the software we stamp. Given the dearth of testers who could serve as mentors in the average organization, we can nonetheless, with conscientious effort, work to grow the herd for ourselves, our craft, and the future of testing.