(An) adaptive spatial-temporal GPU scheduling scheme for multi-domain DNNs services

Abstract

High throughput Deep Neural Networks (DNNs) serving servers are essential for online service applications as deep learning techniques are used in a wider range of applications. Serving DNNs services on serving servers require the key requirement: they need to serve multiple heterogenous DNNs services models that guarantee the service-level objective (SLO) of each model as well as improve the system utilization and improve the system-wide throughput. Therefore, it is required for an GPU scheduler to orchestrate the GPU resources for DNNs models. To address the requirements of DNNs serving servers for multi-domain DNNs services model, this thesis proposes an adaptive combined spatial-temporal GPU scheduling scheme. We first show the existing limitations of current works including both conventional temporal scheduling and spatial scheduling on GPU. By our experiments, we show that the existing spatial scheduling approaches usually lead to high resources reconfiguring time and has not fully utilize the GPU computation resources. To tackle the problem, we propose a combine spatial-temporal GPU scheduling strategy that first deploying an adaptive GPU spatial partitioning strategy to partition the GPU computation to multiple computation parts, which we call GPU partition, to schedule concurrent running models, and secondly define a strategy to share the GPU partition temporally to further enhance the utilization of the GPU system. We investigate the factors that might affect the performance of model when running concurrently and formulate the latency function of spatial-temporal running models and propose a heuristic approach for the problem. Our evaluation shows that the proposed scheme can significantly reduce the resource reconfiguring overhead as well as enhance the system throughput compare to the other baseline works.