The Case for Cooperation between White-Box and Black-Box Test Tools

[article]
Summary:

Although white box and black box testing both produce good results, they are more reliable when done together. Bryan Sullivan lists the strengths and weaknesses of each testing approach and how gray box" testing should be in your testing strategy.

Whether its focus is on detecting functional defects, performance bottlenecks, scalability issues, or security vulnerabilities, virtually every automated Web-application-testing tool on the market today follows one of two separate testing methodologies:  the tool either analyzes the source code of the target application (white-box analysis) or it runs the application and interacts with it through its UI (black-box analysis). The relative merits of each of these approaches are hotly debated within the testing community, but the truth is that neither approach is adequate separate from the other. Black-box analysis is like chocolate and white-box analysis is like peanut butter: each is good on its own, but by combining the two we can create a much more delicious outcome.

The main problem with using each type of tool individually is that one alone doesn't provide adequate code coverage. In this context, code coverage refers to the ability of the testing tool to identify the entire testing surface while thoroughly examining every path of execution of the application. Unless you can be sure that your entire Web application has been tested, you cannot rely on the results of the analysis. This assurance that the entire application has been examined is especially important for security analysis tools. A single functional defect in a page visited by only one user in a thousand may have little impact on the overall perceived quality of the application, but that same security vulnerability on that page may open the entire site to a devastating attack.

Between white-box and black-box analysis, which offers better code coverage? Source analyzers definitely have the edge at finding every page in the application-they already have a list of pages from the source code. They don't need to resort to fragile techniques such as spidering the site or prompting the user to provide a list of pages, as required by black-box analysis tools. Pages like administrator portals that are not linked to any other page in the site may be completely missed by black-box tools but would be found by white-box tools. White-box tools also

About the author

Bryan Sullivan's picture Bryan Sullivan

Bryan Sullivan is a security program manager on the Security Development Lifecycle (SDL) team at Microsoft. He is a frequent speaker at industry events, including Black Hat, BlueHat, and RSA Conference. Bryan is also a published author on Web application security topics. His first book, Ajax Security was published by Addison-Wesley in 2007.

StickyMinds is one of the growing communities of the TechWell network.

Featuring fresh, insightful stories, TechWell.com is the place to go for what is happening in software development and delivery.  Join the conversation now!