On the hyperparameter tuning of a scale-invariant network and its applications

Abstract

Modern deep neural networks are equipped with normalization layers such as batch normalization or layer normalization to enhance and stabilize training dynamics. If a network contains such normalization layers, the optimization objective is invariant to the scale of the neural network parameters. The scale-invariance induces the neural network's output to be only affected by the weights' direction and not the weights' scale. We address the tuning of hyperparameters and their applications in such scale-invariant neural networks. As a first application, hyperparameter tuning in active learning is dealt with. In active learning, the number of labeled training data continues to increase as learning progresses. We propose a weight decay scheduling method suitable for active learning based on the analysis of the relationship between the number of training data and weight decay. We also propose a method of distilling knowledge from the low-performing network used in the previous round and apply it to active learning. We validate our methods on the MNIST, CIFAR-10, and CIFAR-100 datasets using convolutional neural networks of various sizes. Second, we find a common feature of good hyperparameter combinations on such a scale-invariant network, including learning rate, weight decay, number of data samples, and batch size. Our key observation is that hyperparameter setups that lead to good performance show similar degrees of angular update during one epoch. Using a stochastic differential equation, we analyze the angular update and show how each hyperparameter affects it. With this relationship, we can derive a simple hyperparameter tuning method and apply it to an efficient hyperparameter search.