FedHusky: Accelerating Hybrid Federated Learning with Client Hopping

Abstract

Classic federated learning (FL) trains client models in parallel, and its accuracy hinges on each client's local update. But when those updates come from small, highly heterogeneous datasets, parallel FL's performance collapses. Sequential FL alleviates this problem by letting clients update the model in a serial manner. But this sequential execution leaves all but one client idle and wastes computation resources for training. By combining both parallel and sequential FL, hybrid FL partitions clients into groups to run groups in parallel while keeping sequential updates within each group. Motivated by hybrid FL, we propose a novel solution called FedHusky that can fundamentally advance the hybrid FL solution. Most notably, FedHusky employs a novel calendar mechanism at each client to record its busy intervals that have been booked by different working groups. A novel optimization-based approach is employed to generate new groups, aiming at maximizing the number of busy intervals on each client's calendar while avoiding clients being double-booked by different groups at any time. FedHusky also employs a dynamic birth–death process to maintain the active groups during training, enabling a vibrant ecosystem by allowing the membership (clients) in each group to change over time while striving for the maximum number of simultaneous groups. By incorporating a simple delay mechanism, FedHusky can be highly robust to potential estimation errors in key timing parameters, capable of maintaining smooth operation without interruption. Experiments show that under small datasets and high heterogeneity, FedHusky accelerates convergence by at least 3× and increases the average busy ratio by 6.7× when compared with hybrid FL.

Related Papers