Enhancing reliability of deep neural networks through statistics re-weighting

Abstract

In the real world, data often follows a long-tailed distribution. However, in deep learning, if there is imbalance in the training dataset distribution, the model’s generalization performance can sharply decline. Moreover, distributional imbalance may not be limited to a single attribute but can be extend to the joint distribution of multiple attributes. In cases of imbalance in a single attribute, significant performance differences between the tail and head of the distribution can arise. Additionally, when there is imbalance in the joint distribution, fairness issues in the model may arise. To enhance the reliability of deep neural networks, we propose two statistics re weighting methods. First, we suggest a method that utilizes a whitening module to reduce the dependency among representations of each attribute, enabling the training of an unbiased classifier. We propose re-weighted covariance estimation to optimize the trade-off between the utility and fairness of the classifiers. Through this approach, we successfully improved the fairness of the model and enhanced training stability. Second, we propose a method for learning balanced representations for all types of imbalances. By modeling the probability density of features using kernel density estimation and adjusting the weights of the objective function, we improve reliability of representations on various imbalanced datasets.