On Effective Model-Based Intrusion Detection

Abstract

Model-based intrusion detectors restrict program execution to a previously computed model of expected behavior. We consider two classes of attacks against these systems: bypass attacks that evade detection by avoiding the detection system altogether, and transformational attacks that alter a detected attack into a semantically-equivalent attack that goes undetected. Recent detection approaches are problematic and do not effectively address these threats. We see reductions or outright failures in effectiveness and efficiency when systems (1) monitor execution at the library call interface, (2) provide accuracy via inlining of statically-constructed program models, or (3) use simplistic analysis of indirect function calls. Attacks can defeat library-call monitors by directly executing operating system kernel traps. Inlined models grow exponentially large at the trap interface: models for several test programs are 12,000 to 38,000 times larger at the trap interface than at the library call interface. Na??ve indirect call analysis produces models 14 to 177 times larger than models built with in-depth analysis and that are less able to detect attacks. In examining these issues, our aim is to reveal complexities of model-based detection that have not been previously well understood.