Accelerating collective communication for multi-GPU system using detailed multi-GPU simulator

Abstract

Increasing complexity and amount of computation for deep learning are driving the need for multi-GPU systems. To accelerate deep learning training, communication efficiency between GPUs is becoming very critical to determine the overall performance of the system. In this work, we first modeled the state of the art collective communication library in the Multi-GPU Simulator and to accelerate collective communication, we propose an integrated router architecture that uses in-network buffering and in-network computation. With integrated router, we can eliminate the overhead of point-to-point communication that must pass through intermediate nodes, achieve lower memory access latency through in-network buffering, and eliminate resource contention overhead through in-network computation. Finally, we show about 16% improvement in bandwidth utilization and 2.7x faster speed up.