Toward improving the generalization ability of deep features via feature dimension configuration and discriminative feature selection}

Abstract

In this dissertation, we study the generalization ability of the features including deep features or traditional features from various sources. To equip higher generalization ability for features to avoid overfitting on training data, we first propose a novel deep neural network architecture with better generalization ability. Furthermore, we study the generic way to increase the generalization ability for deep neural networks in depth. Finally, we propose post-processing methods for the learned features to achieve better generalization ability.

First part of this dissertation is to develop a novel deep convolutional neural network architecture targeted to state-of-the-art image recognition/classification network. The proposed deep neural network architecture so-called deep pyramidal residual network (PyramidNet) is designed with a novel scheme that increases feature map dimension gradually instead of increasing it sharply in downsampling layers as the network depth goes deeper. Moreover, the zero-padded shortcut is employed in each module to meet the increased feature dimension. It turns out that the generalization ability of the model with the proposed network architecture is significantly improved even with the similar training loss compared to the previous model, and the model can achieve the state-of-the-art results on CIFAR-10 and CIFAR-100 datasets. Furthermore, the proposed network architecture produced the state-of-the-art result on ImageNet-1k dataset and still have comparable performance.

Second, the investigation for a general approach for better generalization ability to equip on a certain network architecture is studied theoretically and empirically in depth. It turns out that monotonically increasing feature dimension can take advantage of avoiding a network architecture to be overfitted, and therefore the trained model would have higher generalization ability. Theoretically, the model with monotonic dimension increasing for features yields a larger loss for each subproblem with respect to each weight layer when training, so it is less likely to be overfitted, and the enlarged final dimension also lowering the intrinsic loss bound for better training. Empirically, these conjectures are successfully proved by evaluating the training and test losses of partial models that use a few successive weight layers from the entire model using all the weight layers. In addition, the empirical studies on CIFAR-10, CIFAR-100, and ImageNet-1k dataset with several existing network architectures show the effectiveness of the monotonic feature dimension increase configuration, and this shows that the proposed configuration would be a prerequisite component of network design.

The final chapter contains a post-processing method, which novel feature refinement methods to handle various features including deep learning features or low-level features in respect of feature selection. The proposed feature selection methods reduce the dimension of the original features (including learned features) by selecting discriminative features regardless of the existence of the ground-truth label information to reduce the dimension of original features. The main idea is to perform orthogonal basis clustering and select features simultaneous. Therefore, the features are selected from the clustered features in the projected space, it turns out that discriminative features could be selected from the projected feature space where less informative features are suppressed. After selecting features, the reduced small-dimensional features while well-preserving the capability of the original features are less prone to overfitting, and this is empirically supported by the state-of-the-art results on 15 public datasets with/without the ground-truth label information compared with more than 10 related works.