Model-agnostic federated learning

Abstract

Deep learning has demonstrated remarkable performance in various fields thanks to a large amount of training data. Meanwhile, the proliferation of edge devices such as mobile and internet-of-things (IoT) devices is on the rise. The edge devices produce substantial amounts of data. There is a growing need for federated learning for on-device AI to train and utilize models using data generated from edge devices. Federated learning is a distributed machine learning framework that leverages the training data and computing resources of distributed users. However, federated learning trains a single global model where the size of models is increasingly becoming larger. Consequently, there may be constrains for edge devices in training a single large model due to their heterogeneous characteristics, including diverse communication network and environments, computing capabilities, and device memory. In this dissertation, the issue is considered by clients having heterogeneous model architectures tailored to the heterogeneous characteristics of clients. Model-agnostic federated learning is a study on federated learning for clients with different model architectures. Two approaches, nested federated learning and generative model-aided federated learning, are introduced for model-agnostic federated learning. Nested federated learning proposes to scale down a global model into submodels where each submodel is subset of the global model and to average the parameters of submodels by nested federated averaging. Generative model-aided federated learning proposes a method of training a single global model to represent data of clients, enabling the training of clients with various model architectures.