Gradient compression with random projection and sequential update method

Abstract

Due to the development of deep learning technology and the increase of the amount of data, distributed training methods for deep learning have attracted a lot of attentions. In a data parallel environment, exchanging calculated gradients between servers is itself a bottleneck. In this paper, we solve this problem by compressing the gradient through random projection and sequentially transmitting it in units of gradients. The methods we proposed show similar compression ratio as the existing methods. In addition, the methods do not require the Top-k algorithm, which is needed in the existing methods, and it is beneficial in terms of time to apply the algorithms. Our algorithms are not limited by the use of parameter server because they do not degrade the compression ratio even when aggregating the calculated gradients at each server. Our algorithm shows a compression ratio from 57× to 919× in the image classification task, although there was some loss of accuracy. In the language model using PTB data, there is no accuracy loss until the compression ratio of 146×.