Localized binary cross-entropy for federated learning

Abstract

Federated Learning is a distributed learning methodology for training a server model across devices that keep their data locally. Recently, Federated Learning has established itself as one of the research fields to address social issues as to privacy infringement. However, a distributed data environment do not follow the identical independent distribution, making it harder for existing methodologies of machine learning to be applied immediately. This data heterogeneity leads to over-fitting on local data in federated learning, which undermines the performance improvement of the server model. In this work, we present Localized Binary Cross-Entropy (LBCE) as a loss function to prevent over-fitting to local data in distributed environments. When a local machine learns a model using a conventional cross-entropy function, it uses error signals for a data class that does not belong to local data. Thus, to limit over-fitting, the LBCE loss function maintains the independence of each class by using a sigmoid function instead of a softmax activation function while regulating signals for data classes that do not belong to local data. LBCE outperformed conventional softmax cross-entropy in various situations of distributed data that did not follow the identical independent distribution.