Software Testing & QA Online Community > Detail: Warm and Fuzzy
|A StickyMinds.com Original|
Warm and Fuzzy
By Bryan Sullivan
Summary: Automated tools are essential to software development. Tools can take the drudgery out of the more tedious development and testing tasks and let us get back to what we love: writing code (or in the tester's case, breaking code). This is especially true for security testing where the goal is not to prove that the software does what it is supposed to do, but rather that it doesn't do what it's not supposed to do. This is a much more difficult task--if not actually an impossible task--but, thankfully, we have some great tools to help us out. In this week's column, Bryan Sullivan covers one of the most valuable of these tools: the fuzzer.
Fuzzers are tools that feed random input to software programs for the purpose of revealing defects in those programs' input-processing logic. For example, if you create a malformed data file and open it in a word processor, does the word processor fail gracefully and inform the user that the file appears to be corrupt? Or does it crash? Or does it start executing the data in the file as if it were assembly code, potentially leading to a complete compromise of the system's security? This scenario is not as farfetched as you might think, if the program contains a buffer-overflow vulnerability. Fuzzers can reveal vulnerabilities like this by automatically generating variations of malformed input and feeding those variations to the program being tested.
You might ask, "How much fuzzing is enough?" It's unlikely that you will find a critical security vulnerability with just a single fuzzed input variation. And, if you do, it's extremely unlikely that you've found the only one in the code! One iteration is clearly not sufficient. On the other hand, you could easily let the fuzzer run for days or weeks, testing millions of iterations, and you still wouldn't be assured that every possibility has been covered. The rule we use in the Microsoft Security Development Lifecycle (SDL) process is that every input must pass 100,000 fuzzing iterations without failure. If a defect is found in the input-processing code, then the module must start the fuzzing process over again after the defect has been fixed and pass another 100,000 iterations without failure.
Another good question is "How much of the data should be malformed?" Let's return to our word processor example. Suppose the word processorís data-file format includes a checksum to ensure that the file hasn't been corrupted or tampered with. Unless the fuzzer is aware of this and modifies the fuzzed data files' checksums accordingly, the vast majority of fuzz iterations it creates will be immediately discarded by the word processor due to checksum failure. This is not only a complete waste of time, but also leads to a false sense of security since the input-handling logic isn't being tested beyond the very shallow first level (the checksum logic).
However, the opposite scenario can be a problem as well. Suppose our fuzzer is aware of the checksum and modifies its fuzzed output data files so that the checksum is always valid. What happens now if there is a defect or security vulnerability in the checksum logic? This logic isn't being tested since the fuzzer has implicitly assumed that it's always correct, an assumption that may not be valid. To put it in higher-level terms, by giving the fuzzer knowledge of the expected input, you're telling it to create input the way the program expects to receive it. But the whole point of fuzzing is to test the program by giving it unexpected input! This is the classic debate between "dumb" fuzzers that have no knowledge of the expected input format and "smart" fuzzers that do. The reality is that neither type of fuzzer is sufficient on its own, and that to achieve the best results both types of fuzzers should be used to test the application.
If you haven't started already, I strongly suggest that you make fuzz testing part of your testing process. The material I've presented here doesn't even scratch the surface of the power and possibilities of fuzzing. If you're interested in learning more, I recommend the book Fuzzing: Brute Force Vulnerability Discovery by Michael Sutton, Adam Greene, and Pedram Amini.
About the Author
Bryan Sullivan is a security program manager on the Security Development Lifecycle (SDL) team at Microsoft. He is a frequent speaker at industry events, including Black Hat, BlueHat, and RSA Conference. Bryan is also a published author on Web application security topics. His first book, Ajax Security was published by Addison-Wesley in 2007.
Back to Top