Enabling future processor-interconnect

Abstract

Multi-socket servers are commonly used in high-performance computing and datacenters. Within each server, a processor-interconnect is used for communication between the different sockets, memory and accelerators and representative examples of such interconnect include Intel QPI and AMD Hypertransport. Recently, it is announced that different technologies of processor-interconnect are under development as processor-interconnect increasingly is taking an important role in modern computer systems. As consideration for designing future processor-interconnect, this thesis explores design issues existing in commercial technology of processor-interconnect with the perspective of security and performance problems of the router architecture. Security vulnerability in hardware level is more critical than in software level as it can often require hardware fix and can lead to serious jeopardy to whole systems. This thesis first presents Processor-Interconnect rootkit (PIkit) that exploits a vulnerable hardware feature within processor-interconnect and shows how a very stealthy rootkit can be achieved compared to previously proposed rootkits. While there has been significant amount of research done on different rootkits, we describe a new type of rootkit that is kernel-independent --i.e., no aspect of the kernel is modified and no code is added to the kernel address space to install the rootkit. In particular, PIkit exploits the DRAM address mapping table structure that determines the destination node ID of a memory request packet in the processor- interconnect. By modifying this mapping table appropriately, PIkit enables access to victim’s memory address region without proper permission. Once PIkit is installed, only user-level code or payload is needed to carry out malicious activities. The malicious payload mostly consists of memory read and/or write instructions that appear like “normal” user-space memory accesses and it becomes very difficult to detect such malicious payload. We describe the design and implementation of PIkit on both an AMD and an Intel x86 multi-socket servers that are commonly used. We discuss different malicious activities possible with PIkit and limitations of PIkit, as well as possible software and hardware solutions to PIkit. Contention to shared resources (e.g., memory, cache and interconnect) in multi-socket systems can result in performance variation (or unfairness) between concurrent applications. This thesis explores the impact of the processor-interconnect on overall performance -- in particular, the performance unfairness caused by processor-interconnect arbitration. It is well known that locally-fair arbitration does not guarantee globally-fair bandwidth sharing as closer nodes receive more bandwidth in a multi-hop network. However, this thesis demonstrates that the opposite can occur in a commodity NUMA server where remote nodes receive higher bandwidth (and perform better). We analyze this problem and iden- tify that this occurs because of external concentration used in router micro-architectures for processor-interconnects without globally-aware arbitration. While accessing remote memory can occur in any NUMA system, performance unfairness (or performance variation) is more critical in cloud computing and virtual machines. We demonstrate how this unfairness creates significant performance variation when a workload is executed on the Xen virtualization platform. To provide fairness, we propose a novel, history-based arbitration that tracks the history of arbitration grants made in the previous history window. A weighted arbitration is done based on the history to provide global fairness. Through simulations, we show our proposed history-based arbitration can provide global fairness and minimize the processor-interconnect performance unfairness at low cost.