Research on federated learning and distillation with implementation in wireless communication environment

Abstract

In this thesis, we investigate the performance of Federated Learning (FL) with Federated Distillation (FD) and Hybrid Federated Distillation (HFD), which is proposed in this these, considering wireless implementations. First of all, we propose the analog and digital transmission schemes how to implement FD and HFD over wireless communication environment. And we compare the performances of FL, FD, and HFD under wireless implementations, even when considering both of uplink and downlink. Furthermore, we modify the analog transmission scheme of FL, especially when distributed stochastic gradient descent (DSGD) is adopted for weight update. The proposed schemes show considerable improvements, especially in the case of non i.i.d. data allocation with low number of allowed channel uses, which can not be supported by the previous analog transmission scheme well. And we propose a update strategy applied in digital transmission scheme for DSGD, which adopts element-wise averaging update scheme. The performance is considerably improved when the scheme is adopted to the state-of-the-art digital transmission for DSGD with error accumulation. Finally, we propose a local weight update scheme for the introduced DSGD implementations. With this scheme, both of digital and analog transmission schemes show considerable improvements compared to the previous DSGD implementations.