Web application vulnerability scanners depend on the relative predictability of Web sites to identify security issues. Using a loose set of rules, scanners function by simulating Web attacks and analyzing the responses for telltale signs of weakness. From experience, we know how a Web site will normally react when there is a security issue present. We know that if sending a Web site certain meta-characters produces a database ODBC error message, a SQL Injection issue has likely been detected. At WhiteHat Security we call these "technical vulnerabilities" and scanners have become fairly proficient at identifying them. But as Web sites become increasingly sophisticated, yesterday's telltale signs are today's false positives. As such, we're not guaranteed that a specific result necessarily indicates that a security issue is present. This has made the automated process of finding simple vulnerabilities hard -- and finding difficult ones impossible.

Consider the following example. If we visit a Web site and are presented with the following URL: http://example/order.asp?item=50&price=300.00

Can we guess what the application order.asp combined with the parameters item and price do? Using intelligence unique to humans, we can quickly deduce their purpose with relative certainty. This is a product ordering application. The item parameter is the particular product we are interested in. In our case, let's say an iPod. The price parameter is the amount we are going to pay for our portable music player. What happens if we changed the price of 300.00 to 100.00? Or 1.00? Does the Web site still sell us the iPod? If so, we can easily understand that the Web site should not have allowed the price alteration. As humans, we possess a natural ability to assess context, and we aptly refer to these types of issues as "logical vulnerabilities," issues that only humans can identify.


Now, if an automated scanner attempted the very same attack in a generic fashion, how would it decide if a custom Web site's response was good or bad? How would it know if the attack worked or was adequately defended? Or what the item and price parameters were supposed to do in the first place? The answer is clear: Scanners cannot reliably make these assumptions. The numbers in the URL easily could have meant something else entirely when presented in a different context. The rules for what is supposed to happen on Web sites are not defined as they are in chess. These decisions require contextual knowledge of the system, plus the ability to "logically" understand any number of previously undefined results.

In mathematics, this very large obstacle is commonly referred to as the undecidable problem. An undecidable problem is a problem that cannot be solved for all cases by any algorithm (or computer program). Chess IS NOT an undecidable problem, since it can be accounted for in all instances at all times. Fully analyzing custom Web application software for vulnerabilities IS an undecidable problem. That's why the game of chess can be fully automated by a computer and identifying vulnerabilities in custom Web applications cannot be. There are unique aspects of the human mind that computers have yet to duplicate. WhiteHat's statistics, based on aggregate data from thousands of assessments, indicate that only about half of the possible Web application security issues can be tested for in a completely automated fashion. The remaining tests for logical issues require the involvement of a Web application security Garry Kasparov.

In a thorough Web application security assessment, potentially hundreds of thousands of customized tests must be performed. By hand, even the world's best experts would never be able to complete this much work in a feasible amount of time.