Data-Free Evaluation of User Contributions in Federated Learning

Abstract

Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.

Meeting Name

19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks, WiOpt (2021: Oct. 18-21, Philadelphia, PA)

Department(s)

Computer Science

Comments

National Science Foundation, Grant 20PJ1407900

Keywords and Phrases

Correlated Agreement; Peer Prediction

International Standard Book Number (ISBN)

978-390317637-9

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2021 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.

Publication Date

21 Oct 2021

Link to Full Text

Share

 
COinS