The importance of data normalization in IPS
To fully comprehend the importance of data normalization in an Intrusion Prevention System, it is first necessary to understand what data normalization is and what it does, how it accomplishes its goal, and why it is so integral to maintaining security against the advanced evasion techniques used today.
The critical importance of data normalization can also be seen while reviewing security failures and fundamental design flaws in many IPS devices that lack such normalization.
Data normalization explained
Data normalization is the process of intercepting and storing incoming data so it exists in one form only. This eliminates redundant data and protects the data’s integrity. The stored, normalized data is protected while any appearance of the data elsewhere is only making a reference to the data that is being stored and protected in the data normalizer.
The normalizer’s job is to patch up the incoming data stream to eliminate the risk of evasion as well as ambiguities. The monitor then views the data in its pure, protected and normalized form. Varying forms of normalization exist on levels of increasing complexity. The complexity is due to the set of requirements that must be met to achieve normalization. The most basic is known as First Normal Form, which is often abbreviated 1NF. It is followed by Second Normal Form, or 2NF, Third Normal Form, or 3NF and can continue increasing in forms and complexity as required or desired.
Normalization benefits
Normalization plays a key role in the security of a network, provided that normalization extends to every protocol layer. One of the major benefits is the forced integrity of the data as data normalization process tends to enhance the overall cleanliness and structure of the data. Normalization significantly contributes to the fortification of a network, especially in light of typical networks’ three main weak points: traffic handling, inspection and detection.
Where many IPS devices go wrong
When it comes to traffic handling, many IPS devices focus on throughput orientation for the most rapid and optimal inline performance. This process, while attractive for its rapidity, makes it impossible for full normalization to take place. The data traffic is then inspected without normalization, offering prime opportunities for infiltration to take place. One may agree that a rapid and optimal output performance is useless if the payload is riddled with malicious invaders.
When many IPS devices do employ normalization, they often rely on shortcuts that only implement partial normalization as well as partial inspection. This leaves gaps in the security and provides optimal opportunity for evasions. TCP segmentation handling is one example of such a process, as it is only executed in chosen protocols or ports and is drastically limited in its execution. Shortcut exploitation is a familiar evasion method and, with the proliferation of IPS devices that fail to perform full normalization, it is likely to remain that way due to its ease of execution.
Many IPS devices fall short in other areas, as well. They often perform only a single layer of analysis, execute traffic modifications and interpretations and rely on inspection of individual segments or pseudo-packets. Their detection methods are based on vulnerability and exploits, banner matching or shell detection. Their updates are generally delayed and their evasion coverage is extremely limited. Evasions can easily exploit the limited inspection scope by spreading attacks over segments or pseudo-packet boundaries.
Packet-oriented pattern matching is insufficient as a means of invasion detection due to the need for a 100% pattern match for blocking or detection. Advanced Evasion Techniques (AETs) possess the ability to utilize a vast multitude of combinations to infiltrate a system, rendering the likelihood of a 100% pattern match for every possible combination nonexistent. It is simply impossible to create enough signatures to be effective.
AETs exploit the weaknesses in the system, often being delivered in a highly liberal manner that a conservatively designed security device is incapable of detecting. In addition to using unusual combinations, AETs also focus on rarely used protocol properties or even create network traffic that disregards strict protocol specifications.
A large number of standard IPS devices fail to detect and block AETs, which have therefore effectively disguised a cyber attack that infiltrates or even decimates the network. Standard methods used to detect and block attacks generally rely on protocol anomalies or violations, which is no longer adequate to match the rapidly changing and adaptable AETs. In fact, the greatest number of anomalies occurs not from attacks, but rather from flawed implementation in regularly used Internet applications.
An additional issue that arises with many IPS devices is the environment in which they are optimized. Optimization typically takes place in a clean or simulated network that has never suffered a complex and highly elusive attack.
Resistance to normalization
Resistance to data normalization does not typically arise from the advanced security it promises, but rather the impact it may have on a network. When the security design flaw is found in hardware-based products, network administrators may resist the upgraded security measure due to the necessity of significant research and development for redesign. Additional memory and CPU capacity are also required to properly implement a data stream inspection that comprehensively protects against AETs.
When vendors decide that the required changes are impossible to implement, they leave their networks highly vulnerable for exploits and attacks. Focusing on the cost of the cleanup required for all infected computers in the network, and the even higher cost of network downtime, can help change the minds of vendors who continue to resist the necessary adaptations.
How the most effective IPS devices use data normalization
Instead of analyzing data as single or combined packets, effective IPS devices analyze data as a normalized stream. Once normalized, the data is sent through multiple parallel and sequential machines. All data traffic should be systematically analyzed by default, regardless of its origins or destination.
The most effective way to detect infiltration is to systematically analyze and decode the data, layer by layer. Normalization must occur at every layer simply because attacks can be hidden at many different layers. In the lower protocol layers, the data stream must be reconstructed in a unique manner. Modifications should generally be very slight or nonexistent, although any fragments or segments containing conflicting and overlapping data should be dropped.
Normalizing traffic in this manner ensures there is a unique way to interpret network traffic passing through the IPS. The data stream is then reassembled for inspection in the upper layers. Inspection of constant data stream in this manner is a must for correcting the flaws and vulnerabilities left open by many IPS devices. This process also removes the possibility of evasion of attacks that span over segment boundaries.
Higher levels are subjected to inspection of separate data streams that are normalized based on the protocol. In compressed HTTP, for instance, the data can be decompressed for inspection. In another example, MSRPC-named pipes using the same SMB connection would be demultiplexed and inspected separately.
Such a thorough and comprehensive data normalization process is the most effective way to protect networks from AETs and other threats that may otherwise disguise themselves to go undetected through standard IPS. The most effective IPS devices will ensure evasions are removed through the normalization process before the data stream is even inspected. This normalization is so successful because it combines a data stream based approach, layered protocol analysis and protocol specific normalization at different levels. It therefore helps fortify a network’s three weakest points and keeps malicious invader’s attacks at bay.