Abstract
Large Language Models (LLMs) are increasingly used in empirical software engineering (ESE) to automate or assist annotation tasks such as labeling commits, issues, and qualitative artifacts. Yet the reliability and reproducibility of such annotations remain underexplored. Existing studies often lack standardized measures for reliability, calibration, and drift, and frequently omit essential configuration details. We argue that LLM-based annotation should be treated as a measurement process rather than a purely automated activity. In this position paper, we outline the Operationalization for LLM-based Annotation Framework (OLAF), a conceptual framework that organizes key constructs: reliability, calibration, drift, consensus, aggregation, and transparency. The paper aims to motivate methodological discussion and future empirical work toward more transparent and reproducible LLM-based annotation in software engineering research.
Recommended Citation
M. M. Imran and T. S. Zaman, "OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering," Proceedings 2026 IEEE ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering Wsese 2026, pp. 33 - 36, Association for Computing Machinery, Jun 2026.
The definitive version is available at https://doi.org/10.1145/3786149.3788306
Department(s)
Computer Science
Publication Status
Free Access
Keywords and Phrases
Annotation; Empirical Software Engineering; LLM; Measurement
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2026 The Authors, All rights reserved.
Publication Date
02 Jun 2026
