article

DUC in context

Authors:

Donna HarmanAuthors Info & Claims

Information Processing and Management: an International Journal, Volume 43, Issue 6

Pages 1506 - 1520

https://doi.org/10.1016/j.ipm.2007.01.019

Published: 01 November 2007 Publication History

Abstract

Recent years have seen increased interest in text summarization with emphasis on evaluation of prototype systems. Many factors can affect the design of such evaluations, requiring choices among competing alternatives. This paper examines several major themes running through three evaluations: SUMMAC, NTCIR, and DUC, with a concentration on DUC. The themes are extrinsic and intrinsic evaluation, evaluation procedures and methods, generic versus focused summaries, single- and multi-document summaries, length and compression issues, extracts versus abstracts, and issues with genre.

References

[1]

Baldwin, B., Donaway, R., Hovy, E., Liddy, E., Mani, I., Marcu, D., et al. (2000). An Evaluation Road Map for Summarization Research. Available from duc.nist.gov/roadmapping.html.

[2]

Blair-Goldensohn, S. (2005). From Definitions to Complex Topics: Columbia University at DUC 2005. Available from duc.nist.gov/pubs/2005papers/columbiau.sasha.pdf.

[3]

Blair-Goldensohn, S., Evans, D., Hatzivassiloglou, V., McKeown, K., Nenkova, A., Passonneau, B., et al. (2004). Columbia University at DUC 2004. Available from duc.nist.gov/pubs/2004papers/columbia2.nenkova.ps.

[4]

Copeck, T., & Szpakowicz, S. (2004). Vocabulary Agreement Among Model Summaries and Source Documents. Available from duc.nist.gov/pubs/2004papers/uottawa.copeck.final.pdf.

[5]

Dang, H.T. (2005). Overview of DUC 2005. Available from duc.nist.gov/pubs/2005papers/OVERVIEW05.pdf.

[6]

D'Avanzo, E., & Magnini, B. (2005). A Keyphrase-Based Approach to Summarization: the LAKE System at DUC-2005. Available from duc.nist.gov/pubs/2005papers/itc-irst.ernesto.pdf.

[7]

Extrinsic Evaluation of Automatic Metrics for Summarization. University of Maryland, College Park.

[8]

Farzindar, A., Rozon, F., & Lapalme, G. (2005). CATS A Topic-Oriented Multi-Document Summarization System at DUC 2005. Available from duc.nist.gov/pubs/2005papers/umontreal.lapalme.pdf.

[9]

Harman, D., & Over, P. (2004). The Effects of Human Variation in DUC Summarization Evaluation. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out. Barcelona, Spain. p. 10-17.

[10]

Harnly, A., Nenkova, A., Passonneau, R., & Rambow, O. (2005). Automation of summary evaluation by the pyramid method. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2005). Borovets, Bulgaria.

[11]

Hovy, E., Lin, C.-Y., Zhou, L., & Fukumoto, J. (2005). Basic Elements. Available from hayden.isi.edu/BE/.

[12]

Jagadeesh, J., Pingali, P., & Varma, V. (2005). A Relevance-Based Language Modeling Approach to DUC 2005. Available from duc.nist.gov/pubs/2005papers/iiit.jagadeesh.pdf.

[13]

Lacatusu, F., Hickl, A., Aarseth, P., & Taylor, L. (2005). Lite-GISTexter at DUC 2005. Available from duc.nist.gov/pubs/2005papers/lcc.finley.pdf.

[14]

Lacatusu, V.F., Parker, P., & Harabagiu, S.M. (2003). Lite-GISTexter:Generating Short Summaries with Minimal Resources. Available from duc.nist.gov/pubs/2003papers/lcc.ps.

[15]

Lin, C.-Y. (2001). Summary Evaluation Environment (SEE). Available from hayden.isi.edu/SEE/.

[16]

Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the ACL-04 Workshop: Text Summarization Branches Out. Barcelona, Spain. p. 74-81.

[17]

Mani, I., Firmin, T., House, D., Klein, G., Sundheim, B., & Hirschman, L. (1999). The TIPSTER SUMMAC Text Summarization Evaluation. In Proceedings of the EACL'99: Ninth Conference of the European Chapter of the Association for Compuational Linguistics. Bergen, Norway. p. 77-85.

Digital Library

[18]

The theory and practice of discourse parsing and summarization. MIT Press, Cambridge, Massachusetts.

Digital Library

[19]

McKeown, K., Barzilay, R., Evans, D., Hatzivassiloglou, V., Klavans, J., Nenkova, A., et al. (2002). Tracking and Summarising News on a Daily Basis with Columbia's Newsblaster. In Proceedings of the Second International Conference on Human Language Technology Research. p. 280-285.

Digital Library

[20]

Melli, G., Wang, Y., Liu, Y., Kashani, M.M., Shi, Z., Gu, B., et al. (2005). Description of SQUASH, the SFU Question Answering Summary Handler for the DUC-2005 Summarization Task. Available from duc.nist.gov/pubs/2005papers/simonfraseru.sarkar.pdf.

[21]

Nakao, Y. (2001). How Small Distinction among Summaries can the Evaluation Method Identify. In Proceedings of the Second NTCIR Workshop Meeting on Evaluation of Chinese and Japanese Text Retrieval and Text Summarization. Tokyo, Japan. p. 235-241.

[22]

Nenkova, A., & Passonneau, R. (2004). Evaluating Content Selection in Summarization: The Pyramid Method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. Boston, MA. p. 145-152.

[23]

Over, P. (2004). An Introduction to DUC 2004 - Intrisic Evaluation of Generic News Text Summarization Systems. Available from duc.nist.gov/pubs/2004slides/duc2004.intro.pdf.

[24]

Proceedings of the Fourth NTCIR Workshop on Research in Information Access Technologies: Information Retrieval, Question-Answering and Summarization. (2004). Available from research.nii.ac.jp/ntcir/workshop/onlineproceedings4/index.html.

[25]

Proceedings of the Second NTCIR Workshop Meeting on Evaluation of Chinese and Japanese Text Retrieval and Text Summarization. (2001). Available from research.nii.ac.jp/ntcir/workshop/onlineproceedings2/index.html.

[26]

Proceedings of the Third NTCIR Workshop on Research in Information Retrieval, Automatic Text Summarization and Question-Answering. (2002). Available from research.nii.ac.jp/ntcir/workshop/onlineproceedings3/index.html.

[27]

Radev, D., Blair-Golddensohn, S., Zhang, Z., & Raghavan, R. (2001). Newsinessence: A System for Domain-Independent, Real-Time Clustering and Multi-Document Summarization. In Proceedings of the First International Conference on Human Language Technology Research. p. 274-277.

Digital Library

[28]

Schilder, F., McCulloh, A., Thomson McInnes, B., & Zhou, A. (2005). TLR at DUC: Tree Similarity. Available from duc.nist.gov/pubs/2005papers/thomson-lr.schilder.pdf.

[29]

Sparck-Jones, K. (1998). Advances in Automatic Text Smmarization. In I. Mani & M.T. Maybury (Eds.), MIT Press, Cambridge, Massachusetts. pp. 1-12.

[30]

Sparck-Jones, K. (2001). Factorial Summary Evaluation. Available from duc.nist.gov/pubs/2001papers/cambridge2.pdf.

[31]

Sparck-Jones, K., van Halteren, H., Moens, M.-F., Lapalme, G., Radev, D., Dorr, B., et al. (2004). DUC Roadmap 2005-2007. Available from duc.nist.gov/RM0507/rm.html.

[32]

Vanderwende, L., Banko, M., & Menezes, A. (2004). Event-Centric Summary Generation. Available from duc.nist.gov/pubs/2004papers/microsoft.banko.pdf.

[33]

Zajic, D., Dorr, B., & Schwartz, R. (2004). BBN/UMD at DUC 2004: Topiary. Available from duc.nist.gov/pubs/2004papers/umaryland.zajic.pdf.

Cited By

Aliakbarpour HManzuri MRahmani A(2024)Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanismMultimedia Tools and Applications10.1007/s11042-023-15589-283:1(733-762)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15589-2
Li YHuang YHuang WWang W(2024)A global and local information extraction model incorporating selection mechanism for abstractive text summarizationMultimedia Tools and Applications10.1007/s11042-023-15274-483:2(4859-4886)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15274-4
Arora A(2022)Ensemble of Support Vector Machine and Ontological Structures to Generate Abstractive Text SummarizationInternational Journal of Information Retrieval Research10.4018/IJIRR.30029412:3(1-24)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.4018/IJIRR.300294
Show More Cited By

Index Terms

DUC in context
1. Applied computing
  1. Document management and text processing
2. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
      2. Language resources

Recommendations

Opinion holder extraction from author and authority viewpoints
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Opinion holder extraction research is important for discriminating between opinions that are viewed from different perspectives. In this paper, we describe our experience of participation in the NTCIR-6 Opinion Analysis Pilot Task by focusing on opinion ...
Sentiment diversification for short review summarization
WI '17: Proceedings of the International Conference on Web Intelligence

With the abundance of reviews published on the Web about a given product, consumers are looking for ways to view major opinions that can be presented in a quick and succinct way. Reviews contain many different opinions, making the ability to show a ...
Multilingual opinion holder identification using author and authority viewpoints

Opinion holder identification research is important for discriminating between opinions that are viewed from different perspectives. We propose a new opinion holder identification method that is based on a differentiation between the author and ...

Comments

Information & Contributors

Information

Published In

cover image Information Processing and Management: an International Journal

Information Processing and Management: an International Journal Volume 43, Issue 6

November, 2007

412 pages

ISSN:0306-4573

Issue’s Table of Contents

Copyright © © 2007.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 November 2007

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aliakbarpour HManzuri MRahmani A(2024)Automatic text summarization using deep reinforced model coupling contextualized word representation and attention mechanismMultimedia Tools and Applications10.1007/s11042-023-15589-283:1(733-762)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15589-2
Li YHuang YHuang WWang W(2024)A global and local information extraction model incorporating selection mechanism for abstractive text summarizationMultimedia Tools and Applications10.1007/s11042-023-15274-483:2(4859-4886)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s11042-023-15274-4
Arora A(2022)Ensemble of Support Vector Machine and Ontological Structures to Generate Abstractive Text SummarizationInternational Journal of Information Retrieval Research10.4018/IJIRR.30029412:3(1-24)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.4018/IJIRR.300294
Aliakbarpour HManzuri MRahmani A(2022)Improving the readability and saliency of abstractive text summarization using combination of deep neural networks equipped with auxiliary attention mechanismThe Journal of Supercomputing10.1007/s11227-021-03950-x78:2(2528-2555)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1007/s11227-021-03950-x
Žagar ARobnik-Šikonja M(2022)Cross-lingual transfer of abstractive summarizer to less-resource languageJournal of Intelligent Information Systems10.1007/s10844-021-00663-858:1(153-173)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1007/s10844-021-00663-8
Neri-Mendoza VLedeneva YGarcía-Hernández RHernández-Castañeda Á(2022)Multi-document Text Summarization Based on Genetic Algorithm and the Relevance of Sentence FeaturesPattern Recognition10.1007/978-3-031-07750-0_24(255-265)Online publication date: 22-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-07750-0_24
Zhu CYang ZGmyr RZeng MHuang XDiaz FShah CSuel TCastells PJones RSakai T(2021)Leveraging Lead Bias for Zero-shot Abstractive News SummarizationProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462846(1462-1471)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3462846
Rojas-Simón JLedeneva YGarcía-Hernández R(2021)Evaluation of text summaries without human references based on the linear optimization of content metrics using a genetic algorithmExpert Systems with Applications: An International Journal10.1016/j.eswa.2020.113827167:COnline publication date: 1-Apr-2021
https://dl.acm.org/doi/10.1016/j.eswa.2020.113827
Takase SKobayashi SLarochelle HRanzato MHadsell RBalcan MLin H(2020)All word embeddings from one embeddingProceedings of the 34th International Conference on Neural Information Processing Systems10.5555/3495724.3496042(3775-3785)Online publication date: 6-Dec-2020
https://dl.acm.org/doi/10.5555/3495724.3496042
Neri-Mendoza VLedeneva YGarcía-Hernández RPinto DSingh VPerez F(2020)Unsupervised extractive multi-document text summarization using a genetic algorithmJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-17990039:2(2397-2408)Online publication date: 1-Jan-2020
https://dl.acm.org/doi/10.3233/JIFS-179900
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents