Article

Free access

Statistical significance of MUC-6 results

Author:

Nancy ChinchorAuthors Info & Claims

MUC6 '95: Proceedings of the 6th conference on Message understanding

Pages 39 - 43

https://doi.org/10.3115/1072399.1072404

Published: 06 November 1995 Publication History

PDF eReader

Abstract

The results of the MUC-6 evaluation must be analyzed to determine whether close scores significantly distinguish systems or whether the differences in those scores are a matter of chance. In order to do such an analysis, a method of computer intensive hypothesis testing was developed by SAIC for the MUC-3 results and has been used for distinguishing MUC scores since that time. The implementation of this method for the MUC evaluations was first described in [1] and later the concepts behind the statistical model were explained in a more understandable manner in [2]. This paper gives the results of the statistical testing for the three MUC-6 tasks where a single metric could be associated with a system's performance.

References

[1]

Chinchor, N., Hirschman, L., and D. Lewis (1993) "Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3)" Computational Linguistics 19(3).

Digital Library

Google Scholar

[2]

Chinchor, N. (1992). "The Statistical Significance of the MUC-4 Results" Proceedings of the Fourth Message Understanding Conference (MUC-4). Morgan Kaufmann, Publishers. San Mateo, CA.

Digital Library

Google Scholar

[3]

Noreen, W. (1989) Computer Intensive Methods for Testing Hypotheses: An Introduction. John Wiley & Sons.

Google Scholar

[4]

Van Rijsbergen, C. J. (1979) Information Retrieval. London: Butterworths.

Digital Library

Google Scholar

Cited By

View all

Zhao SNg HJoshi AHuang CJurafsky D(2010)Maximum metric score training for coreference resolutionProceedings of the 23rd International Conference on Computational Linguistics10.5555/1873781.1873928(1308-1316)Online publication date: 23-Aug-2010
https://dl.acm.org/doi/10.5555/1873781.1873928
Kummerfeld JRoesner JDawborn THaggerty JCurran JClark SHajič J(2010)Faster parsing by supertagger adaptationProceedings of the 48th Annual Meeting of the Association for Computational Linguistics10.5555/1858681.1858717(345-355)Online publication date: 11-Jul-2010
https://dl.acm.org/doi/10.5555/1858681.1858717
Branavan SChen HEisenstein JBarzilay R(2009)Learning document-level semantic properties from free-text annotationsJournal of Artificial Intelligence Research10.5555/1622716.162273134:1(569-603)Online publication date: 1-Apr-2009
https://dl.acm.org/doi/10.5555/1622716.1622731
Show More Cited By

Recommendations

Overview of results of the MUC-6 evaluation
MUC6 '95: Proceedings of the 6th conference on Message understanding

The latest in a series of natural language processing system evaluations was concluded in October 1995 and was the topic of the Sixth Message Understanding Conference (MUC-6) in November. Participants were invited to enter their systems in as many as ...
The statistical significance of the MUC-5 results
MUC5 '93: Proceedings of the 5th conference on Message understanding

The statistical significance of the results of the MUC-5 evaluation is determined using a computer-intensive method of hypothesis testing known as approximate randomization. The exact method is described in detail in [1] and [2] and has been used as the ...
Overview of results of the MUC-6 evaluation
TIPSTER '96: Proceedings of a workshop on held at Vienna, Virginia: May 6-8, 1996

The latest in a series of natural language processing system evaluations was concluded in October 1995 and was the topic of the Sixth Message Understanding Conference (MUC-6) in November. Participants were invited to enter their systems in as many as ...

Comments

Information & Contributors

Information

Published In

MUC6 '95: Proceedings of the 6th conference on Message understanding

November 1995

353 pages

ISBN:1558604022

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 06 November 1995

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
281
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)12

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhao SNg HJoshi AHuang CJurafsky D(2010)Maximum metric score training for coreference resolutionProceedings of the 23rd International Conference on Computational Linguistics10.5555/1873781.1873928(1308-1316)Online publication date: 23-Aug-2010
https://dl.acm.org/doi/10.5555/1873781.1873928
Kummerfeld JRoesner JDawborn THaggerty JCurran JClark SHajič J(2010)Faster parsing by supertagger adaptationProceedings of the 48th Annual Meeting of the Association for Computational Linguistics10.5555/1858681.1858717(345-355)Online publication date: 11-Jul-2010
https://dl.acm.org/doi/10.5555/1858681.1858717
Branavan SChen HEisenstein JBarzilay R(2009)Learning document-level semantic properties from free-text annotationsJournal of Artificial Intelligence Research10.5555/1622716.162273134:1(569-603)Online publication date: 1-Apr-2009
https://dl.acm.org/doi/10.5555/1622716.1622731
Chen HBranavan SBarzilay RKarger DOstendorf MCollins MNarayanan SOard D(2009)Global models of document structure using latent permutationsProceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics10.5555/1620754.1620808(371-379)Online publication date: 31-May-2009
https://dl.acm.org/doi/10.5555/1620754.1620808
Gamon MAue ARingger E(2005)Automatic identification of sentiment vocabularyProceedings of the ACL Workshop on Feature Engineering for Machine Learning in Natural Language Processing10.5555/1610230.1610241(57-64)Online publication date: 29-Jun-2005
https://dl.acm.org/doi/10.5555/1610230.1610241
Zhou GSu JIsabelle P(2002)Named entity recognition using an HMM-based chunk taggerProceedings of the 40th Annual Meeting on Association for Computational Linguistics10.3115/1073083.1073163(473-480)Online publication date: 6-Jul-2002
https://dl.acm.org/doi/10.3115/1073083.1073163
Vieira RPoesio M(2000)An empirically based system for processing definite descriptionsComputational Linguistics10.1162/08912010075010594826:4(539-593)Online publication date: 1-Dec-2000
https://dl.acm.org/doi/10.1162/089120100750105948
Bikel DSchwartz RWeischedel R(1999)An Algorithm that Learns What‘s in a NameMachine Language10.1023/A:100755822112234:1-3(211-231)Online publication date: 1-Feb-1999
https://dl.acm.org/doi/10.1023/A%3A1007558221122

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

Overview of results of the MUC-6 evaluation

The statistical significance of the MUC-5 results

Overview of results of the MUC-6 evaluation

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations