Optimizing the aggregate throughput of concurrent deep learning jobs on a shared cluster

Abstract

The explosive popularity of deep learning (DL) has led to the evolution of deep learning frameworks. Unfortunately, despite the need for running multiple deep learning jobs on a GPU shared cluster environment, current cloud schedulers are often insufficient to schedule them efficiently. Managing resources for deep learning models without enough information or expertise results in poor performance in scalability and adversely affects the overall cluster performance. In this paper, we present Max-Speedup, a job scheduling policy of multi-tenant deep learning jobs on a shared GPU cluster. We address two main challenges, 1) precise estimation of training throughput to analyse resource-performance trade-off of a deep learning model and 2) efficient scheduling policy for multi-tenant deep learning jobs on a shared GPU cluster. We tackle these problems by estimating the finish time of parameter synchronization and maximizinig aggregate speedup by exploiting performance-resource trade-offs of DL jobs. Our evaluation shows that Max-Speedup improves the average job completion time by 3x over SRTF while it reduces makespan by up to 26.9x.