Abstract
The extraction of patient signs and symptoms recorded as free text in electronic health records is critical for precision medicine. Once extracted, signs and symptoms can be made computable by mapping to signs and symptoms in an ontology. Extracting signs and symptoms from free text is tedious and time-consuming. Prior studies have suggested that inter-rater agreement for clinical concept extraction is low. We have examined inter-rater agreement for annotating neurologic concepts in clinical notes from electronic health records. After training on the annotation process, the annotation tool, and the supporting neuro-ontology, three raters annotated 15 clinical notes in three rounds. Inter-rater agreement between the three annotators was high for text span and category label. A machine annotator based on a convolutional neural network had a high level of agreement with the human annotators but one that was lower than human inter-rater agreement. We conclude that high levels of agreement between human annotators are possible with appropriate training and annotation tools. Furthermore, more training examples combined with improvements in neural networks and natural language processing should make machine annotators capable of high throughput automated clinical concept extraction with high levels of agreement with human annotators.
Recommended Citation
C. Oommen et al., "Inter-rater Agreement For The Annotation Of Neurologic Signs And Symptoms In Electronic Health Records," Frontiers in Digital Health, vol. 5, article no. 1075771, Frontiers Media, Jan 2023.
The definitive version is available at https://doi.org/10.3389/fdgth.2023.1075771
Department(s)
Chemistry
Keywords and Phrases
annotation; clinical concept extraction; electronic health records; inter-rater agreement; natural language processing; neural networks; phenotype; signs and symptoms
International Standard Serial Number (ISSN)
2673-253X
Document Type
Article - Journal
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2023 The Authors, All rights reserved.
Creative Commons Licensing
This work is licensed under a Creative Commons Attribution 4.0 License.
Publication Date
01 Jan 2023
Comments
U.S. Department of Veterans Affairs, Grant BX000467