(An) efficient partitioning and scheduling algorithm for GPUs in designing machine learning inference server

Abstract

Today's cloud vendors offer Machine Learning as a Service (MLaaS). Unlike the training process, inference does not require high computational power, and inference using GPUs does not fully utilize the computational power of the device. The recently proposed GPU allows providers to partition single GPU into units of a size suitable for the degree of user's request and provides the ability to lower their Total Cost of Ownership (TCO) through increased computational utilization. This dissertation proposes a method of improving the compute utilization through heterogeneity of the multi-GPU server. The sophisticated partitioning algorithm proposed (PARIS) heterogeneizes inference servers based on the model and the characteristics of the environment, and guarantees Service Level Agreement (SLA) through the appropriate scheduling method (ELSA). The proposed partitioning and scheduling algorithm achieves an maximum 17.4x and 1.8x improvement in latency and throughput, respectively.