Test-time augmentation methods for image classification and robustness to common noise via image resolution modification

Abstract

Image classification has been a core task to measure the degree of image understanding. The advanced architectures and training algorithms in image classification have been transferred into various areas. Although the success guarantees trustful classification performance for clean images, it is easily degraded on noise. Although many studies have proposed novel methods to overcome the corruptions, it has been a challenging problem to improve performances for both classification and robustness at the same time.

This dissertation aims to improve performances for both classification and robustness to common noise at the same time. In order to achieve the aim, we propose simple yet effective test-time augmentation methods for single and multiple crop evaluations in both tasks. All proposed methods consider resizing mainly as a sub-policy of test-time augmentation methods.

In the first proposed methods, inspired by \cite{FixRes}, the scaling factors are designed without learning to balance low and high resolutions based on the observation that low (high) resolution is effective on robustness (classification) but contrarily they are much ineffective to the other tasks. In particular, for high resolution images like ImageNet dataset, scale factors are determined based on theoretical analysis for both crop evaluations. Experiments demonstrate that our first methods improve both performances of image classification and robustness to noise, while showing superior or at least comparable results to the standard and learning-based methods. In particular, for high resolution images, our multi-crop method shows superior or at least comparable performances only with 3 crops.

In the second proposed methods, we improve the first proposed multi-crop method for high resolution images by adjusting a degree of confidence of prediction scores. Based on the observation that lower (higher) resolution images generally provide higher (lower) probabilities for top-1 predictions, softmax probabilities of the lowest resolution image are enhanced except for top-1. Also, activations of an inner layer of a pretrained convolutional neural network are adjusted towards the same objective. With the methods, the softmax score of each crop provides a broad variety of information that can be helpful to correct a wrong prediction when averaging multi-crop predictions especially. In Experiments, our second methods improve the performance of the first proposed one. In particular, our improved multi-crop method, with only 2 crops, shows comparable performances to the previous method with 3 crops for prediction.

Besides, we investigate the possibility of extensions of the proposed methods. The problems are to extend our methods to additional types of corruptions, pretrained CNNs which violate the assumption of the proposed methods, and self-supervised learning algorithms. Simple experiments demonstrate the possibility of the extensions to those problems.

Notably, in practice, our methods are not only adapted on most pretrained models but they are also easily applied with few changes in an off-the-shelf way. Hence, we expect the proposed methods to yield immediate improvement.