(A) preemptible neural processing unit architecture and its applicability for QoS-aware scheduling

Abstract

To meet the high demands for Deep Neural Network (DNN) computation, major cloud vendors offer Machine Learning (ML) acceleration as a service. Due to the high compute requirement of DNN application, GPUs and chips specially designed for DNN computation called Neural Processing Units (NPUs) are generally used for the computation. Service providers utilize multi-tenancy by co-locating multiple DNN models inside a single accelerator to achieve high throughput and reduce the Total Cost of Ownership (TCO). However, current NPUs lack the ability to preempt an on-going task and cannot provide fast response time to high priority requests leading to Service Level Agreement (SLA) violation. To improve the Quality of Service (QoS) of ML services, this dissertation explores three possible preemption mechanisms for NPUs and proposes a scheduling algorithm (PREMA) working on top of the preemptible NPU. Overall, the proposed scheduler achieves an average of 7.8$\times$, 1.4$\times$, and 4.8$\times$ improvement in latency, throughput, and SLA satisfaction, respectively.