Countdown: mitigation of software exploitation using usage count-based system call filter

Abstract

System calls have always been abused for many software exploitations. Although modern operating systems have mechanisms to filter the unnecessary system calls, their approach supports only policies of allowing or rejecting, requires manual configuration, and often fails to depend against sophisticated exploits that utilize only permitted system calls. This research suggests a more fine-granular methodology to restrict critical but rarely utilized system calls based on the number of usages. The filtering rules for system calls are generated by dynamically profiling the target applications and are enforced during runtime. The prototype of this research analyzed real-world applications to determine usage limits for each system call. The results show that it can reduce attack surfaces without harming the programs' functionalities, incurring similar performance overhead as a basic filtering mechanism Linux provides. Since the usage limits are automatically found for each system call, it can be practically applied without manual inspection of software source code.