Automatic parallelization for efficient distributed deep learning

Abstract

Deep learning is increasingly popular as it is applied to many problems of various fields with excellent performance. Unfortunately, it often takes a large amount of time to train a practical deep model for real-world quality. While distributed deep learning that harnesses multiple GPUs in parallel can potentially reduce training time significantly, it is not only onerous but also error-prone to manually write the logic for distributed training of each model. Some research works have tackled the problem by proposing algorithms that automatically parallelize deep learning models given a single-GPU model. However, they often support only basic functions for distributed training, and misses a number of important factors for efficient distributed deep learning.In this paper, we propose an automatic parallelization for efficient distributed training beyond basic parallelization of a deep learning model. First, we minimize the overhead in switching the device set to a new configuration, which allows fast re-adjustment of aggregate resource of a GPU cluster. Second, it enables training with a large batch size even with a single GPU, which leads to better utilization of cluster resource. Finally, it realizes near-uniform load balancing, reducing the variation of processing latency and better resource utilization. Evaluation shows that our auto-parallelization expands a device set up to 6 times faster and shrink a device set almost instantly, and improves the scalability up to 2.97 times, compared to existing methods.