Finite state machines for monitoring software-based attacks from untrusted kernel

Abstract

Most existing software-based system attacks (hereinafter referred to as software attacks) assume that the kernel is secure and that the user runs a malicious program. Under these assumptions, defenses against software attacks are usually designed based on trust in the kernel. However, if the kernel is attacked or turns into an attacker, the victims cannot use the defense method based on trust in the kernel. Hardware monitoring can protect users even if the kernel is not trusted. This thesis investigates state-of-the-art software attacks which attack the kernel or the kernel attacks the trusted execution environment as an attacker. This thesis aims to analyze software-based attacks in units of instructions and registers to establish monitoring rules for hardware monitor.