research-article

A Study of Realtime Summarization Metrics

Authors:

Matthew Ekstrand-Abueg,

Richard McCreadie,

Fernando DiazAuthors Info & Claims

CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

Pages 2125 - 2130

https://doi.org/10.1145/2983323.2983653

Published: 24 October 2016 Publication History

Abstract

Unexpected news events, such as natural disasters or other human tragedies, create a large volume of dynamic text data from official news media as well as less formal social media. Automatic real-time text summarization has become an important tool for quickly transforming this overabundance of text into clear, useful information for end-users including affected individuals, crisis responders, and interested third parties. Despite the importance of real-time summarization systems, their evaluation is not well understood as classic methods for text summarization are inappropriate for real-time and streaming conditions.

The TREC 2013-2015 Temporal Summarization (TREC-TS) track was one of the first evaluation campaigns to tackle the challenges of real-time summarization evaluation, introducing new metrics, ground-truth generation methodology and dataset. In this paper, we present a study of TREC-TS track evaluation methodology, with the aim of documenting its design, analyzing its effectiveness, as well as identifying improvements and best practices for the evaluation of temporal summarization systems.

References

[1]

J. Allan, editor. Topic Detection and Tracking: Event-based Information Organization. Inf. Retrieval. 2002.

Digital Library

[2]

J. Allan, R. Gupta, and V. Khandelwal. Temporal summaries of new topics. In Proc of SIGIR, 2001.

Digital Library

[3]

O. Alonso and R. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Proc. of ECIR, 2011.

Digital Library

[4]

G. Baruah, A. Roegiest, and M. D. Smucker. The effect of expanding relevance judgements with duplicates. In Proc. of SIGIR, 2014.

Digital Library

[5]

B. Carterette, E. Gabrilovich, V. Josifovski, and D. Metzler. Measuring the reusability of test collections. In Proc. of WSDM, 2010.

Digital Library

[6]

Y. Chen, N. J. Conroy, and V. L. Rubin. News in an online world: the need for an automatic crap detector. In Proc. of ASIS&T, 2015.

[7]

H. T. Dang and K. Owczarzak. Overview of the TAC 2008 update summarization task. In Proc. of TAC, 2008.

[8]

M. Dostert and D. Kelly. Users' stopping behaviors and estimates of recall. In Proc. of SIGIR, 2009.

Digital Library

[9]

Q. Guo, F. Diaz, and E. Yom-Tov. Updating users about time critical events. In Inf. Retrieval. Springer, 2013.

Digital Library

[10]

M. Imran, C. Castillo, F. Diaz, and S. Vieweg. Processing social media messages in mass emergency: A survey. ACM Comput. Surv., July 2015.

Digital Library

[11]

B. Keegan, D. Gergle, and N. Contractor. Hot off the wiki: Structures and dynamics of wikipedia's coverage of breaking news events. American Behavioral Scientist, 2013.

[12]

C.-Y. Lin. ROUGE: a Package for Automatic Evaluation of Summaries. In Proc. of ACL, 2004.

[13]

C.-Y. Lin and F. Och. Looking for a few good metrics: Rouge and its evaluation. In NTCIR Workshop, 2004.

[14]

H. P. Luhn. A business intelligence system. IBM J. Res. Dev., 1958.

Digital Library

[15]

R. Mccreadie, C. Macdonald, and I. Ounis. Identifying top news using crowdsourcing. Inf. Retrieval, 2013.

Digital Library

[16]

A. Nenkova and K. McKeown. Automatic summarization. Foundations and Trends in Information Retrieval, 2011.

[17]

P. Thomas and D. Hawking. Evaluation by comparing result sets in context. In Proc. of CIKM, 2006.

Digital Library

[18]

Y. Wang, G. Sherman, J. Lin, and M. Efron. Assessor differences and user preferences in tweet timeline generation. In Proc. of SIGIR, 2015.

Digital Library

[19]

E. Yilmaz, J. A. Aslam, and S. Robertson. A new rank correlation coefficient for information retrieval. In Proc. of SIGIR, 2008.

Digital Library

[20]

Y. Zhang, J. Callan, and T. Minka. Novelty and redundancy detection in adaptive filtering. In Proc. of SIGIR, 2002.

Digital Library

Cited By

Widyassari ARustad SShidik GNoersasongko ESyukur AAffandy ASetiadi D(2022)Review of automatic text summarization techniques & methodsJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2020.05.00634:4(1029-1046)Online publication date: Apr-2022
https://doi.org/10.1016/j.jksuci.2020.05.006
McCreadie RMacdonald COunis ICollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Automatic Ground Truth Expansion for Timeline EvaluationThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210034(685-694)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210034
Baruah GMcCreadie RLin JLim EWinslett MSanderson MFu ASun JCulpepper SLo EHo JDonato DAgrawal RZheng YCastillo CSun ATseng VLi C(2017)A Comparison of Nuggets and Clusters for Evaluating Timeline SummariesProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3133000(67-76)Online publication date: 6-Nov-2017
https://dl.acm.org/doi/10.1145/3132847.3133000
Show More Cited By

Index Terms

A Study of Realtime Summarization Metrics
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
      1. Temporal reasoning
  2. Modeling and simulation
    1. Simulation types and techniques
      1. Real-time simulation
2. Information systems
  1. Information retrieval

Recommendations

Intertopic information mining for query-based summarization

In this article, the authors address the problem of sentence ranking in summarization. Although most existing summarization approaches are concerned with the information embodied in a particular topic (including a set of documents and an associated ...
Topic and sentiment aware microblog summarization for twitter
Abstract
Recent advances in microblog content summarization has primarily viewed this task in the context of traditional multi-document summarization techniques where a microblog post or their collection form one document. While these techniques already ...
Using topic themes for multi-document summarization

The problem of using topic representations for multidocument summarization (MDS) has received considerable attention recently. Several topic representations have been employed for producing informative and coherent summaries. In this article, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

October 2016

2566 pages

ISBN:9781450340731

DOI:10.1145/2983323

General Chairs:
Snehasis Mukhopadhyay
Indiana University Purdue University Indianapolis, USA
,
ChengXiang Zhai
University of Illinois at Urbana-Champaign, USA
,
Program Chairs:
Elisa Bertino
Purdue University
,
Fabio Crestani
University of Lugano
,
Javed Mostafa
University of North Carolina
,
Jie Tang
Tsinghua University
,
Luo Si
Alibaba Group Inc & Purdue University
,
Xiaofang Zhou
University of Queensland
,
Yi Chang
Yahoo Research
,
Yunyao Li
IBM Research - Almaden
,
Parikshit Sondhi
WalmartLabs

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM'16

Sponsor:

CIKM'16: ACM Conference on Information and Knowledge Management

October 24 - 28, 2016

Indiana, Indianapolis, USA

Acceptance Rates

CIKM '16 Paper Acceptance Rate 160 of 701 submissions, 23%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
263
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Widyassari ARustad SShidik GNoersasongko ESyukur AAffandy ASetiadi D(2022)Review of automatic text summarization techniques & methodsJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2020.05.00634:4(1029-1046)Online publication date: Apr-2022
https://doi.org/10.1016/j.jksuci.2020.05.006
McCreadie RMacdonald COunis ICollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Automatic Ground Truth Expansion for Timeline EvaluationThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210034(685-694)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210034
Baruah GMcCreadie RLin JLim EWinslett MSanderson MFu ASun JCulpepper SLo EHo JDonato DAgrawal RZheng YCastillo CSun ATseng VLi C(2017)A Comparison of Nuggets and Clusters for Evaluating Timeline SummariesProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3133000(67-76)Online publication date: 6-Nov-2017
https://dl.acm.org/doi/10.1145/3132847.3133000
Ghosh SGhosh KGanguly DChakraborty TJones GMoens M(2017)ECIR 2017 Workshop on Exploitation of Social Media for Emergency Relief and Preparedness (SMERP 2017)ACM SIGIR Forum10.1145/3130332.313033851:1(36-41)Online publication date: 2-Aug-2017
https://dl.acm.org/doi/10.1145/3130332.3130338

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents