Towards Inclusive Fairness Evaluation Via Eliciting Disagreement Feedback From Non-expert Stakeholders

Abstract

Traditional algorithmic fairness notions rely on label feedback, which can only be elicited from expert critics. However, in most practical applications, several non-expert stakeholders also play a major role in the system and can have distinctive opinions about the decision making philosophy. For example, in kidney placement programs, transplant surgeons are very wary about accepting kidney offers for black patients due to genetic reasons. However, non-expert stakeholders in kidney placement programs (e.g. patients, donors and their family members) may misinterpret such decisions from the perspective of social discrimination. This paper evaluates group fairness notions from the viewpoint of non-expert stakeholders, who can only provide binary agreement/disagreement feedback regarding the decision in context. Specifically, two types of group fairness notions have been identified: (i) definite notions (e.g. calibration), which can be evaluated exactly using disagreement feedback, and (ii) indefinite notions (e.g. equal opportunity) which suffer from uncertainty due to lack of label feedback. In the case of indefinite notions, bounds are presented based on disagreement rates, and an estimate is constructed based on established bounds. The efficacy of all our findings are validated empirically on real human feedback dataset.

Department(s)

Computer Science

International Standard Book Number (ISBN)

978-303174629-1

International Standard Serial Number (ISSN)

1865-0937; 1865-0929

Document Type

Article - Conference proceedings

Document Version

Citation

File Type

text

Language(s)

English

Rights

© 2025 Springer, All rights reserved.

Publication Date

01 Jan 2025

Link to Full Text

Share

 
COinS