(A) novel optimization algorithm with orthogonality for deep neural networks inspired by feedback integrators

Abstract

The optimization with orthogonality has been proven to be useful in training deep neural networks (DNNs). To impose orthogonality on DNNs, existing algorithms either utilize hard constraints or soft constraints. However, the methods using hard constraints are computationally expensive, and those based on soft constraints can hardly maintain the orthogonality during the whole training process. To this end, we propose a novel method, named Feedback Gradient Descent (FGD), that induces orthogonality based on the simple Euler discretization of a continuous-time dynamical system on the tangent bundle of the Stiefel manifold, showing high efficiency and stability simultaneously. Rather than using time-consuming structure-preserving discretization methods such as variational or symplectic integrators, we employ the framework of feedback integrators for the discretization. Namely, a continuous-time dynamical system is constructed in a Euclidean space containing the tangent bundle of the Stiefel manifold such that the tangent bundle becomes a local exponential attractor of the system. Since the system is in a Euclidean space, the stability of the tangent bundle is carried over to its discretized system with any off-the-shelf discretization method such as Euler, yielding the FGD algorithm that is fast in speed and stable in the preservation of the tangent bundle of the Stiefel manifold. We conduct extensive image classification experiments on popular benchmark datasets, e.g., CIFAR-10/100 and ImageNet, using various models, e.g., WideResNet and ResNet, where FGD comprehensively outperforms the existing state-of-the-art methods in terms of accuracy, efficiency, and stability.