Improving Federated Learning Through Low-Entropy Client Sampling Based on Learned High-Level Features

Data heterogeneity impacts the performance of Federated Learning (FL) by introducing training noise. Although representative client sampling can help mitigate the issue, it remains challenging to implement without compromising data privacy. This work introduces a new method to address the problem by proposing an affordable blind (privacy preserving) clustering mechanism for conducting stratified client sampling. Inspired by the 'dialect quiz', we propose a 'response test' to cluster clients whose models have learned similar high-level features. This approach facilitates representative client sampling without the need for direct access to client data. We demonstrate empirically that our method yields client samples with low relative entropy with respect to the global data distribution, indicating increased representativeness. Convergence experiments reveal that applying our method significantly improves the convergence and accuracy of the global model compared to strong baselines like SCAFFOLD and FL-CIR. Additionally, the reduced number of training rounds required to achieve target accuracy leads to decreased communication overhead and computational expense, making our approach promising for practical FL implementations.