[PDF][PDF] A comparison of rankings produced by summarization evaluation measures

RL Donaway, KW Drummey… - NAACL-ANLP 2000 …, 2000 - aclanthology.org
RL Donaway, KW Drummey, LA Mather
NAACL-ANLP 2000 Workshop: Automatic Summarization, 2000aclanthology.org
Abstract Summary evaluation measures produce a ranking of all possible extract summaries
of a document., Recall-based evaluation measures, which depend on costly human-
generated ground truth summaries, produce uncorrelated rankings when ground truth is
varied. This paper proposes using sentence-rankbased and content-based measures for
evaluating extract summaries, and compares these with recallbased evaluation measures.
Content-based measures increase the correlation of rankings induced by synonymous …
Abstract
Summary evaluation measures produce a ranking of all possible extract summaries of a document., Recall-based evaluation measures, which depend on costly human-generated ground truth summaries, produce uncorrelated rankings when ground truth is varied. This paper proposes using sentence-rankbased and content-based measures for evaluating extract summaries, and compares these with recallbased evaluation measures. Content-based measures increase the correlation of rankings induced by synonymous ground truths, and exhibit other desirable properties.
aclanthology.org