期刊
IEEE TRANSACTIONS ON SERVICES COMPUTING
卷 15, 期 2, 页码 614-626出版社
IEEE COMPUTER SOC
DOI: 10.1109/TSC.2021.3109910
关键词
Collaborative work; Synchronization; Convergence; Training; Servers; Computational modeling; Probabilistic logic; Federated learning; edge computing; distributed computing; barrier control
资金
- TravelSky Technology Limited
This paper presents a new barrier control technique called Probabilistic Synchronous Parallel (PSP) to address the challenges of synchronization methods and barrier control methods in federated learning. The research shows that PSP improves convergence speed and scalability, achieving a good balance between system efficiency and model accuracy.
With the massive amount of data generated from mobile devices and the increase of computing power of edge devices, the paradigm of Federated Learning has attracted great momentum. In federated learning, distributed and heterogeneous nodes collaborate to learn model parameters. However, while providing benefits such as privacy by design and reduced latency, the heterogeneous network present challenges to the synchronisation methods, or barrier control methods, used in training, regarding system progress and model convergence etc. The design of these barrier mechanisms is critical for the performance and scalability of federated learning systems. We propose a new barrier control technique called Probabilistic Synchronous Parallel (PSP). In contrast to existing mechanisms, it introduces a sampling primitive that composes with existing barrier control mechanisms to produce a family of mechanisms with improved convergence speed and scalability. Our proposal is supported with a convergence analysis of PSP-based SGD algorithm. In practice, we also propose heuristic techniques that further improve the efficiency of PSP. We evaluate the performance of proposed methods using the federated learning specific FEMNSIT dataset. The evaluation results show that PSP can effectively achieve good balance between system efficiency and model accuracy, mitigating the challenge of heterogeneity in federated learning.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据