Data-Free Evaluation of User Contributions in Federated Learning
Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.
H. Lv et al., "Data-Free Evaluation of User Contributions in Federated Learning," Proceedings of the 19th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (2021, Philadelphia, PA)), Institute of Electrical and Electronics Engineers (IEEE), Oct 2021.
The definitive version is available at https://doi.org/10.23919/WiOpt52861.2021.9589136
19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks, WiOpt (2021: Oct. 18-21, Philadelphia, PA)
Keywords and Phrases
Correlated Agreement; Peer Prediction
International Standard Book Number (ISBN)
Article - Conference proceedings
© 2021 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
21 Oct 2021