research-article

Free access

DUC 2005: evaluation of question-focused summarization systems

Author:

Hoa Trang DangAuthors Info & Claims

SumQA '06: Proceedings of the Workshop on Task-Focused Summarization and Question Answering

Pages 48 - 55

Published: 23 July 2006 Publication History

PDF eReader

Abstract

The Document Understanding Conference (DUC) 2005 evaluation had a single user-oriented, question-focused summarization task, which was to synthesize from a set of 25--50 documents a well-organized, fluent answer to a complex question. The evaluation shows that the best summarization systems have difficulty extracting relevant sentences in response to complex questions (as opposed to representative sentences that might be appropriate to a generic summary). The relatively generous allowance of 250 words for each answer also reveals how difficult it is for current summarization systems to produce fluent text from multiple documents.

References

[1]

Enrique Amigo, Julio Gonzalo, Victor Peinado, Anselmo Penas, and Felisa Verdejo. 2004. An empirical study of information synthesis tasks. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 207--214, Barcelona, Spain.

Digital Library

Google Scholar

[2]

Donna Harman and Paul Over. 2004. The effects of human variation in duc summarization evaluation. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, pages 10--17, Barcelona, Spain.

Google Scholar

[3]

Eduard Hovy, Chin-Yew Lin, and Liang Zhou. 2005. Evaluating duc 2005 using basic elements. In Proceedings of the Fifth Document Understanding Conference (DUC), Vancouver, Canada.

Google Scholar

[4]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out, pages 74--81, Barcelona, Spain.

Google Scholar

[5]

Rebecca J. Passonneau, Ani Nenkova, Kathleen McKeown, and Sergey Sigelman. 2005. Applying the pyramid method in duc 2005. In Proceedings of the Fifth Document Understanding Conference (DUC), Vancouver, Canada.

Google Scholar

[6]

Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 316--323, Tampere, Finland, August.

Digital Library

Google Scholar

Cited By

View all

Perera PKosseim L(2014)Evaluation of Sentence Compression Techniques against Human PerformanceProceedings of the 15th International Conference on Computational Linguistics and Intelligent Text Processing - Volume 840410.1007/978-3-642-54903-8_46(553-565)Online publication date: 6-Apr-2014
https://dl.acm.org/doi/10.1007/978-3-642-54903-8_46
Das PSrihari RFu Y(2011)Simultaneous joint and conditional modeling of documents tagged from two perspectivesProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063773(1353-1362)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063773
Belz AKow EViethen JGatt A(2010)Generating referring expressions in contextEmpirical methods in natural language generation10.5555/1880370.1880390(294-327)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.5555/1880370.1880390
Show More Cited By

DUC 2005: evaluation of question-focused summarization systems
1. Computing methodologies
  1. Artificial intelligence
2. Hardware
  1. Power and energy
    1. Power estimation and optimization

Recommendations

The DUC summarization evaluations
HLT '02: Proceedings of the second international conference on Human Language Technology Research

There has been a long history of research in text summarization by both the text retrieval and the natural language processing communities, but evaluation of this research has always presented problems. In 2001 NIST launched a new text summarization ...
DUC in context

Recent years have seen increased interest in text summarization with emphasis on evaluation of prototype systems. Many factors can affect the design of such evaluations, requiring choices among competing alternatives. This paper examines several major ...
The TREC 2005 robust track

The TREC robust retrieval track explores methods for improving the consistency of retrieval technology by focusing on poorly performing topics. The retrieval task in the track is a traditional ad hoc retrieval task where the evaluation methodology ...

Comments

Information & Contributors

Information

Published In

SumQA '06: Proceedings of the Workshop on Task-Focused Summarization and Question Answering

July 2006

70 pages

ISBN:1932432795

Program Chairs:
Tat-Seng Chua
National University of Singapore (Singapore)
,
Jade Goldstein
US Department of Defense (USA)
,
Simone Teufel
Cambridge University (UK)
,
Lucy Vanderwende
Microsoft Research (USA)

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 23 July 2006

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
334
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)7

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Perera PKosseim L(2014)Evaluation of Sentence Compression Techniques against Human PerformanceProceedings of the 15th International Conference on Computational Linguistics and Intelligent Text Processing - Volume 840410.1007/978-3-642-54903-8_46(553-565)Online publication date: 6-Apr-2014
https://dl.acm.org/doi/10.1007/978-3-642-54903-8_46
Das PSrihari RFu Y(2011)Simultaneous joint and conditional modeling of documents tagged from two perspectivesProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063773(1353-1362)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063773
Belz AKow EViethen JGatt A(2010)Generating referring expressions in contextEmpirical methods in natural language generation10.5555/1880370.1880390(294-327)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.5555/1880370.1880390
Belz AKow EKelleher JNamee Bvan der Sluis I(2010)Comparing rating scales and preference judgements in language evaluationProceedings of the 6th International Natural Language Generation Conference10.5555/1873738.1873743(7-15)Online publication date: 7-Jul-2010
https://dl.acm.org/doi/10.5555/1873738.1873743
Tutos AMollá D(2010)A study on the use of search engines for answering clinical questionsProceedings of the Fourth Australasian Workshop on Health Informatics and Knowledge Management - Volume 10810.5555/1862303.1862314(61-68)Online publication date: 1-Jan-2010
https://dl.acm.org/doi/10.5555/1862303.1862314

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

The DUC summarization evaluations

DUC in context

The TREC 2005 robust track

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations