RAPIDO: a rejuvenating adaptive PID-type optimiser for deep neural networks

Abstract

The authors present a novel gradient descent algorithm called RAPIDO for deep learning. It adapts over time and performs optimisation using current, past and future information similar to the PID controller. The proposed method is suited for optimising deep neural networks that consist of activation functions such as sigmoid, hyperbolic tangent and ReLU functions because it can adapt appropriately to sudden changes in gradients. They experimentally study the authors' method and show the performance results by comparing with other methods on the quadratic objective function and the MNIST classification task. The proposed method shows better performance than the other methods.