Specific initialization method for endomorphism-like operators in residual networks

Abstract

Residual neural networks have changed the way we approach machine learning, and deep learning specifically, giving a new scope of training, and allowing deeper neural networks to exist. However, whereas their contribution are large, there are still many unknowns on the reason why networks designed this way perform so well.

On this paper, we provide a set of analysis of residual networks in opposition to their non residual counterpart, arguing that one of the strength of this type of architecture is to give a nice initialization space that allows the network to efficiently train. This analysis leads us toward a new type of initialization that specifically target endomorphism-type layers, which we explain how to select on different network architectures. We give results of zero and low variance initialization for those targeted layer in many different training setting. We then provide a set of analysis of the possible better understanding of neural networks this approach could give, using a recently introduced similarity measure designed for neural networks post activation comparison, Central Kernel Alignment. Using this similarity measure, we conclude that one of the major merits of this approach is helping de-correlating targeted layers during training.