Abstract
Entity-aware document retrieval uses query-associated entities as ranking signals, assuming that semantically relevant entities are also useful retrieval signals. We show this assumption is insufficient - and explain why. Unlike terms, which are ground-truth observations, entity links are hypotheses produced by an imperfect linker: an entity can be topically central yet provide no discriminative signal if the linker fires indiscriminately across relevant and non-relevant documents. We formalize this as a distinction between Conceptual Entity Relevance (CER) - whether an entity is topically related to a query - and Observable Entity Relevance (OER) - whether its observed presence in a collection discriminates relevant from non-relevant documents. Across four collections and annotation sources including human entity judgments, CER and OER exhibit near-chance agreement (k ≈ 0), while OER operationalizations agree substantially (k ≈ 0.5), confirming CER as the systematic outlier. CER-based supervision selects topically plausible but weakly discriminative entities, pruning fewer than 4% of non-relevant documents on some collections. Aligning supervision with OER improves non-relevant pruning by up to 10x and open-world MAP by 0.051 over BM25. Our findings motivate a shift from conceptual to observable notions of entity relevance in entity-aware retrieval.
Recommended Citation
U. K. Ghosh and S. Chatterjee, "Entity Labels Are Not Entity Signals: A Framework For Observable Relevance In Document Re-Ranking," Ictir 2026 Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval, pp. 118 - 128, Association for Computing Machinery, Jul 2026.
The definitive version is available at https://doi.org/10.1145/3805713.3820411
Department(s)
Computer Science
Publication Status
Open Access
Keywords and Phrases
entity signal diagnosis; entity-aware retrieval; observable entity relevance; retrieval evaluation
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2026 The Author(s), All rights reserved.
Creative Commons Licensing
This work is licensed under a Creative Commons Attribution 4.0 License.
Publication Date
24 Jul 2026
