On the Analysis of Byte-Granularity Heap Randomization

Abstract

Heap randomization, in general, has been a well-trodden area; however, the efficacy of byte-granularity randomization has never been fully explored as misalignment raises various concerns. Modern heap exploits often abuse the determinism in word alignment, and modern CPU architecture better supports unaligned access (since Nehalem). Based on such new developments, we conduct an in-depth analysis of evaluating the efficacy of byte-granularity heap randomization in three folds: (i) security effectiveness, (ii) performance impact, and (iii) compatibility analysis to measure deployment cost. Security discussion is based on 20 CVE case studies. To measure performance details, we conduct cycle-level microbenchmarks and report that the performance cost is highly concentrated to edge cases depending on the L1-cache line. Based on such analysis, we design and implement an allocator suited for byte-granularity heap randomization. On the negative side, our analysis suggests that byte-granularity heap randomization has high deployment cost due to various implementation conflicts. We enumerate the problematic compatibility issues using Coreutils, Nginx, and ChakraCore benchmarks.