(A) study on design, implementation, and optimizations of external hardware-based kernel integrity monitor

Abstract

Kernel-level malwares poses a grave threat to system security since they operate with the highest privilege within the victimized system. Researchers have sought to establish various Trusted Execution Environ- ment (TEE)that enables reliable detection of such vicious malwares. External hardware-based kernel monitoring has been overlooked in the midst of a plethora of hypervisor-based kernel integrity solutions, and the existing hardware-based monitor is limited to periodic snapshot checks on kernel static regions. In this dissertation, We explore the external hardware-based kernel monitoring approach by presenting the new design,implementations and monitoring techniques of external hardware-based kernel integrity monitors. In this dissertation, we present a event-triggered external hardware-based kernel integrity monitor. With our prototypes, we introduce a new monitoring technique called snoop-based monitoring method- ology against kernel’s both static and dynamic regions. With the snooper module’s event-triggered monitoring we show that our approach can detect transient attacks on kernel static regions that can be missed by snapshot-based monitoring techniques. Furthermore, we present a more advanced monitor- ing platform built around the snooping technique called KI-Mon. KI-Mon advances the snoop-based monitoring with hardware-based memory value whitelisting to selectively allow only certain values to be written on to the monitored memory region. KI-Mon also executes developer-defined kernel integrity verifying routines upon detection of designated memory transaction on the system bus. In this way, we allow developers to write custom kernel integrity verifying (with the KI-Mon API), that are triggered when pre-defined memory transaction pattern appears on the KI-Mon platform. We also present external monitoring aware Linux kernel memory allocator modifications that sim- plifies a number of formidable challenges that are faced by external monitors in general. We slightly modify the kernel’s layered memory allocators (Buddy and SLAB allocator) to congregate only the mon- itored objects into a single contiguous memory region, namely the ZONE MONITORED. In this way, the burdensome task of constantly tracing the monitored kernel object type is eliminated. Furthermore, we report a critical security vulnerability in our external monitor design that may allow adversary to evade the monitoring from the external monitors. The Address Translation Redirection Attack (ATRA) redefines the virtual to physical address translation to relocate the monitored objects and create rogue copies of the objects. We provide a rigorous analysis on Linux kernel’s memory mapping properties. Based on this, we introduce a memory mapping verification scheme for external monitors. In all, this dissertation introduces a series of explorations on external hardware-based kernel integrity monitors. We explain the design, implementation and evaluation of our external monitors and monitoring techniques built on top of them. We hope that our work shed a light on the under explored external kernel integrity monitor approach, and prove to be a feasible solution in kernel integrity monitoring.