article

Methodological Review: Annotating temporal information in clinical narratives

Authors:

Anna Rumshisky,

Ozlem UzunerAuthors Info & Claims

Journal of Biomedical Informatics, Volume 46

Pages S5 - S12

https://doi.org/10.1016/j.jbi.2013.07.004

Published: 01 December 2013 Publication History

Abstract

Temporal information in clinical narratives plays an important role in patients' diagnosis, treatment and prognosis. In order to represent narrative information accurately, medical natural language processing (MLP) systems need to correctly identify and interpret temporal information. To promote research in this area, the Informatics for Integrating Biology and the Bedside (i2b2) project developed a temporally annotated corpus of clinical narratives. This corpus contains 310 de-identified discharge summaries, with annotations of clinical events, temporal expressions and temporal relations. This paper describes the process followed for the development of this corpus and discusses annotation guideline development, annotation methodology, and corpus quality.

References

[1]

Meystre, S., Savova, G., Kipper-Schuler, K. and Hurdle, J., Extracting information from textual documents in the electronic health record: a review of recent research. Yearb Med Inform. v35. 128-144.

[2]

Sun W, Rumshisky A, Uzuner O. Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. Journal of the American Medical Informatics Association 2013.

[3]

Augusto, J.C., Temporal reasoning for decision support in medicine. Artif Intell Med. v33 i1. 1-24.

[4]

Stacey, Michael and McGregor, Carolyn, Temporal abstraction in intelligent clinical data analysis: a survey. Artif Intell Med. v39 i1. 1-24.

[5]

Denny Joshua C, Peterson Josh F, Choma Neesha N, Hua Xu, Miller Randolph A, Bastarache Lisa, et al. Extracting timing and status descriptors for colonoscopy testing from electronic medical records. J Am Med Inform Assoc 2010;17(4):383-8.

[6]

Liu Mei, Jiang Min, Kawai Vivian K, Stein Charles M, Roden Dan M, Denny Joshua C, et al. Modeling drug exposure data in electronic medical records: an application to warfarin. In: AMIA annual symposium proceedings, volume 2011. American Medical Informatics Association; 2011. p. 815.

[7]

Irvine Ann K, Haas Stephanie W, Sullivan Tessa. Tn-ties: A system for extracting temporal information from emergency department triage notes. In: AMIA Annual symposium proceedings, volume 2008. American Medical Informatics Association; 2008. p. 328.

[8]

Zhou, Li, Parsons, Simon and Hripcsak, George, The evaluation of a temporal reasoning system in processing clinical discharge summaries. J Am Med Inform Assoc. v15 i1. 99-106.

[9]

Clark Kimberly K, Sharma Deepak K, Chute Christopher G, Tao Cui. Application of a temporal reasoning framework tool in analysis of medical device adverse events. In: AMIA annual symposium proceedings, volume 2011. American Medical Informatics Association; 2011. p. 1366.

[10]

Li, M. and Patrick, J., Extracting temporal information from electronic patient records. AMIA Annu Sympos Proc. v2012. 542

[11]

Pustejovsky, J., Castano, J., Ingria, R., Sauri, R., Gaizauskas, R. and Setzer, A., Timeml: Robust specification of event and temporal expressions in text. New Direct Question Answer. v3. 28-34.

[12]

Augusto, J.C., The logical approach to temporal reasoning. Artif Intell Rev. v16 i4. 301-333.

[13]

Mani I, Pustejovsky J, Gaizauskas R. The language of time: a reader. Oxford University Press; 2005.

[14]

Allen JF. An interval-based representation of temporal knowledge. In: Proc. 7th international joint conference on artificial intelligence. Canada: Vancouver; 1981. p. 221-6.

Digital Library

[15]

Mani, I., Pustejovsky, J. and Sundheim, B., Introduction to the special issue on temporal information processing. ACM Trans Asian Lang Inform Process (TALIP). v3 i1. 1-10.

[16]

Pustejovsky J, Hanks P, Sauri R, See A, Gaizauskas R, Setzer A et al. The TimeBank corpus. In: Corpus linguistics, volume 2003; 2003. p. 40.

[17]

Verhagen M, Gaizauskas R, Schilder F, Hepple M, Katz G, Pustejovsky J. Semeval-2007 task 15: Tempeval temporal relation identification. In: Proceedings of the 4th international workshop on semantic evaluations; 2007. p. 75-80.

[18]

Verhagen M, Sauri R, Caselli T, Pustejovsky J. Semeval-2010 task 13: Tempeval-2. In: Proceedings of the 5th international workshop on semantic evaluation. Association for, Computational Linguistics; 2010. p. 57-62.

[19]

UzZaman N, Llorens H, Allen J, Derczynski L, Verhagen M, Pustejovsky J. Tempeval-3: Evaluating events, time expressions, and temporal relations. arXiv, preprint arXiv:1206.5333; 2012.

[20]

Roberts A, Gaizauskas R, Hepple M, Demetriou G, Guo Y, Setzer A et al. Semantic annotation of clinical text: the clef corpus. In: Proceedings of building and evaluating resources for biomedical text mining: workshop at LREC; 2008.

[21]

Galescu L, Blaylock N. A corpus of clinical narratives annotated with temporal information. In: Proceedings of the 2nd ACM SIGHIT symposium on international health informatics. ACM; 2012. p. 715-20

Digital Library

[22]

Castaño J, Gaizauskas R, Ingria B, Katz G, Knippen B, Littman J et al. 1.2.1 a formal specification language for events and temporal expressions; 2005. p. 10.

[23]

UzZaman N, Allen JF. Trips and trios system for tempeval-2: Extracting temporal information from text. In: Proceedings of the 5th international workshop on semantic evaluation. Association for, Computational Linguistics; 2010. p. 276-83

Digital Library

[24]

Llorens H, Saquete E, Navarro B. Tipsem (English and Spanish): evaluating crfs and semantic roles in tempeval-2. In: Proceedings of the 5th international workshop on semantic, evaluation; 2010. p. 284-91.

Digital Library

[25]

Graff, D. and Cieri, C., English Gigaword. 2003. Linguistic Data Consortium, Philadelphia.

[26]

Uzuner, O., Solti, I. and Cadag, E., Extracting medication information from clinical text. J Am Med Inform Assoc. v17 i5. 514-518.

[27]

Savova G, Bethard S, Styler W, Martin J, Palmer M, Masanz et al. Towards temporal relation discovery from the clinical narrative. In: AMIA annual symposium proceedings, volume 2009. American Medical Informatics Association; 2009. p. 568.

[28]

Jordan, P.W., Mowery, D.L., Wiebe, J. and Chapman, W.W., Annotating conditions in clinical narratives to support temporal classification. Proc Am Med Inform Assoc Sympos. v2010. 1005

[29]

Bramsen P, Deshpande P, Lee YK, Barzilay R. Finding temporal order in discharge summaries. In: AMIA annual symposium proceedings, volume 2006. American Medical Informatics Association; 2006. p. 81.

[30]

Uzuner, O., South, B., Shen, S. and DuVall, S., 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. v18 i5. 552-556.

[31]

Evaluating the state of the art in coreference resolution for electronic medical records. J Am Med Inform Assoc. v19 i5. 786-791.

[32]

Temporal histories of your medical events, thyme.

[33]

Stubbs A. Mae and mai: lightweight annotation and adjudication tools. In: Proceedings of the 5th linguistic annotation workshop. Association for, Computational Linguistics; 2011. p. 129-33.

[34]

Fleiss Joseph L, Levin Bruce, Paik Myunghee Cho. Statistical methods for rates and proportions. John Wiley & Sons; 2013.

[35]

Hripcsak, George and Rothschild Adam, S., Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc. v12 i3. 296-298.

[36]

Setzer A, Gaizauskas R, Hepple M. Using semantic inferences for temporal annotation comparison. In: Proceedings of the fourth international workshop on inference in computational semantics (ICOS-4); 2003. p. 25-6

[37]

UzZaman N, Allen J. Temporal evaluation. In: Proceedings of the 49th annual meeting of the association for computational linguistics. Human Language Technologies; 2011. p. 351-6.

Digital Library

Cited By

Frattallone-Llado GKim JCheng CSalazar DEdakalavan SWeiss J(2024)Using Multimodal Data to Improve Precision of Inpatient Event TimelinesAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2238-9_25(322-334)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1007/978-981-97-2238-9_25
Campos RJorge AJatowt ABhatia SLitvak M(2024)The 7th International Workshop on Narrative Extraction from Texts: Text2Story 2024Advances in Information Retrieval10.1007/978-3-031-56069-9_52(391-397)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56069-9_52
Lee JWanyan TChen QKeenan TGlicksberg BChew ELu ZWang FPeng Y(2022)Predicting Age-related Macular Degeneration Progression with Longitudinal Fundus Images Using Deep LearningMachine Learning in Medical Imaging10.1007/978-3-031-21014-3_2(11-20)Online publication date: 18-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-21014-3_2
Show More Cited By

Index Terms

Methodological Review: Annotating temporal information in clinical narratives
1. Applied computing
  1. Life and medical sciences
    1. Health care information systems
2. Hardware
  1. Power and energy
    1. Power estimation and optimization
      1. Platform power issues

Index terms have been assigned to the content through auto-classification.

Recommendations

MCN: A comprehensive corpus for medical concept normalization
Graphical abstract

Display Omitted
Highlights
- Clinical concept normalization corpus with a comprehensive coverage is made available.
Abstract
Normalization of clinical text involves linking different ways of talking about the same clinical concept to the same term in the standardized vocabulary. To date, very few annotated corpora for normalization have been available, and ...
Methodological Review: Temporal reasoning with medical data-A review with emphasis on medical natural language processing

Temporal information is crucial in electronic medical records and biomedical information systems. Processing temporal information in medical narrative data is a very challenging area. It lies at the intersection of temporal representation and reasoning (...
Automatic transformation from TIDES to TimeML annotation

Until recently, most systems performing temporal extraction and reasoning from text have focused on recognizing and normalizing temporal expressions alone, for which the TIDES annotation scheme has been adopted. Temporal awareness of a text, however, ...

Comments

Information & Contributors

Information

Published In

Copyright © Elsevier Inc. © 2013.

Publisher

Elsevier Science

San Diego, CA, United States

Publication History

Published: 01 December 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Frattallone-Llado GKim JCheng CSalazar DEdakalavan SWeiss J(2024)Using Multimodal Data to Improve Precision of Inpatient Event TimelinesAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2238-9_25(322-334)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1007/978-981-97-2238-9_25
Campos RJorge AJatowt ABhatia SLitvak M(2024)The 7th International Workshop on Narrative Extraction from Texts: Text2Story 2024Advances in Information Retrieval10.1007/978-3-031-56069-9_52(391-397)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56069-9_52
Lee JWanyan TChen QKeenan TGlicksberg BChew ELu ZWang FPeng Y(2022)Predicting Age-related Macular Degeneration Progression with Longitudinal Fundus Images Using Deep LearningMachine Learning in Medical Imaging10.1007/978-3-031-21014-3_2(11-20)Online publication date: 18-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-21014-3_2
Yadav SRamteke PEkbal ASaha SBhattacharyya P(2020)Exploring Disorder-Aware Attention for Clinical Event ExtractionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/337232816:1s(1-21)Online publication date: 17-Apr-2020
https://dl.acm.org/doi/10.1145/3372328
de Azevedo RRodrigues Jda Silva Reis MMoro CParaiso E(2018)Temporal Tagging of Noisy Clinical Texts in Brazilian PortugueseComputational Processing of the Portuguese Language10.1007/978-3-319-99722-3_24(231-241)Online publication date: 24-Sep-2018
https://dl.acm.org/doi/10.1007/978-3-319-99722-3_24
Wang WKreimeyer KWoo EBall RFoster MPandey AScott JBotsis T(2016)A new algorithmic approach for the extraction of temporal associations from clinical narratives with an application to medical product safety surveillance reportsJournal of Biomedical Informatics10.1016/j.jbi.2016.06.00662:C(78-89)Online publication date: 1-Aug-2016
https://dl.acm.org/doi/10.1016/j.jbi.2016.06.006
Madkour MBenhaddou DTao C(2016)Temporal data representation, normalization, extraction, and reasoningComputer Methods and Programs in Biomedicine10.1016/j.cmpb.2016.02.007128:C(52-68)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1016/j.cmpb.2016.02.007
Savkov ACarroll JKoeling RCassell J(2016)Annotating patient clinical records with syntactic chunks and named entitiesLanguage Resources and Evaluation10.1007/s10579-015-9330-750:3(523-548)Online publication date: 1-Sep-2016
https://dl.acm.org/doi/10.1007/s10579-015-9330-7
Wenzina RKaiser KKostkova PGrasso F(2015)GOALSProceedings of the 5th International Conference on Digital Health 201510.1145/2750511.2750520(121-128)Online publication date: 18-May-2015
https://dl.acm.org/doi/10.1145/2750511.2750520

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents