Overcoming Forgetting, in Federated Learning on Non-IID Data.
We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning.
We tackle the problem of Federated Learning in the non i.i.d. case, in which local models drift apart, inhibiting learning. Building on an analogy with Lifelong Learning, we adapt a solution for catastrophic forgetting to Federated Learning. We add a penalty term to the loss function, compelling all local models to converge to a shared optimum. We show that this can be done efficiently for communication (adding no further privacy risks), scaling with the number of nodes in the distributed setting. Our experiments show that this method is superior to competing ones for image recognition on the MNIST dataset.
Distributed training on edge devices: Batch Normalisation.
An Edgify Research Team Publication. In the first post of this series, we presented two basic approaches to distributed…
Distributed Training on Edge Devices. Large Batch vs. Federated Learning.
An Edgify Research Team Publication, this is the first, introductory post, in our three-part algorithmic series on real-world distributed training.
Distributed Training on Edge Devices: Communication compression.
An Edgify.ai Research Team Publication. In the first post of this series, we presented two basic approaches to distributed training on edge…