Bayesian learning and unlearning in distributed wireless network

Abstract

Bayesian federated learning (FL) offers a principled framework for the definition of collaborative training algorithms that are able to quantify epistemic uncertainty and to produce trustworthy decisions. Upon the completion of collaborative training, an agent may decide to exercise her legal ``right to be forgotten'', which calls for her contribution to the jointly trained model to be deleted and discarded. This paper studies FL and unlearning in a decentralized network within a Bayesian framework. It specifically develops federated variational inference solutions based on the decentralized solution of local free energy minimization problems within exponential-family models and on local gossip-driven communication.

Then, this paper proposes to leverage the flexibility of non-parametric Bayesian approximate inference to develop a novel Bayesian federated unlearning method, referred to as \emph{Forget-Stein Variational Gradient Descent (Forget-SVGD)}. Variational \emph{particle-based} Bayesian learning methods have the advantage of not being limited by the bias affecting more conventional parametric techniques. Upon the completion of FL, Forget-SVGD carries out local SVGD updates at the agents whose data need to be ``unlearned''. The proposed method is validated via performance comparisons with non-parametric schemes that train from scratch by excluding data to be forgotten, as well as with existing parametric Bayesian unlearning methods.

Finally, conventional frequentist FL schemes are known to yield overconfident decisions. Bayesian FL addresses this issue by allowing agents to process and exchange uncertainty information encoded in distributions over the model parameters. However, this comes at the cost of a larger per-iteration communication overhead. This paper investigates whether Bayesian FL can still provide advantages in terms of calibration when constraining communication bandwidth. We present compressed particle-based Bayesian FL protocols for FL and federated ``unlearning" that apply quantization and sparsification across multiple particles. The experimental results confirm that the benefits of Bayesian FL are robust to bandwidth constraints.