short-paper

The BOLT IR Test Collections of Multilingual Passage Retrieval from Discussion Forums

Authors:

SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

Pages 713 - 716

https://doi.org/10.1145/2911451.2914674

Published: 07 July 2016 Publication History

Get Access

Abstract

This paper describes a new test collection for passage retrieval from multilingual, informal text. The task being modeled is that of a monolingual English-speaking user who wishes to search discussion forum text in a foreign language. The system retrieves relevant short passages of text and presents them to the user, translated into English. The test collection contains more than 2 billion words of discussion thread text, 250 queries representing complex informational search needs, and manual relevance judgments of forum post passages, pooled from real systems. This information retrieval test collection is the first to combine multilingual search, passage retrieval, and informal online genre text.

References

[1]

J. Allan. HARD track overview in TREC 2005: High accuracy retrieval from documents. In Proceedings of TREC 2005, pages 52--68. NIST, 2006.

Google Scholar

[2]

BOLT program home page. http://www.darpa.mil/program/broad-operational-language-translation, retrieved on Feb 1, 2016.

Google Scholar

[3]

K. Griffitt and S. Strassel. The query of everything: Developing open-domain, natural language queries for BOLT information retrieval. In Proceedings of LREC 2016, Portoro\vz, Slovenia, 2016.

Google Scholar

[4]

M. Kaszkiel and J. Zobel. Passage retrieval revisited. In Proceedings of SIGIR 1997, Philadelphia, PA, 1997.

Digital Library

Google Scholar

[5]

J. Medero, K. Maeda, S. Strassel, and C. Walker. An efficient approach for gold-standard annotation: Decision points for complex tasks. In Proceedings of LREC 2006, Genoa, Italy, 2006.

Google Scholar

[6]

T. Sakai and N. Kando. On information retrieval metrics designed for evaluation with incomplete relevance assessments. Information Retrieval, 11(5):447--470, March 2008.

Digital Library

Google Scholar

[7]

S. Tellex, B. Katz, J. Lin, A. Fernandes, and G. Martin. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of SIGIR 2003, Toronto, Canada, 2003.

Digital Library

Google Scholar

[8]

E. M. Voorhees and D. K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.

Digital Library

Google Scholar

[9]

W. Xu, R. Grishman, and L. Zhao. Passage retrieval for information extraction using distant supervision. In Proceedings of the 5th International Joint Conference on Natural Language Processing, Chiang Mai, Thailand, 2011.

Google Scholar

[10]

E. Yilmaz, E. Kanoulas, and J. A. Aslam. A simple and efficient sampling method for estimating AP and NDCG. In Proceedings of SIGIR 2008, Singapore, 2008.

Digital Library

Google Scholar

Cited By

View all

Dietz LHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)A Workbench for Autograding Retrieve/Generate SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657871(1963-1972)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657871
Lima de Oliveira LRomeu RMoreira VDiaz FShah CSuel TCastells PJones RSakai T(2021)REGIS: A Test Collection for Geoscientific Documents in PortugueseProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463256(2363-2368)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463256
Rücklé ASwarnkar KGurevych I(2019)Improved Cross-Lingual Question Retrieval for Community Question AnsweringThe World Wide Web Conference10.1145/3308558.3313502(3179-3186)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313502
Show More Cited By

Index Terms

The BOLT IR Test Collections of Multilingual Passage Retrieval from Discussion Forums
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections
    2. Specialized information retrieval
      1. Structure and multilingual text search
        Multilingual and cross-lingual retrieval

Recommendations

Enhancing relevance models with adaptive passage retrieval
ECIR'08: Proceedings of the IR research, 30th European conference on Advances in information retrieval

Passage retrieval and pseudo relevance feedback/query expansion have been reported as two effective means for improving document retrieval in literature. Relevance models, while improving retrieval in most cases, hurts performance on some heterogeneous ...
MT-based Japanese-Enlish cross-language IR experiments using the TREC test collections
IRAL '00: Proceedings of the fifth international workshop on on Information retrieval with Asian languages

This paper evaluates the effectiveness of MT-based Japanese-English CLIR using a subcollection of the TREC test collections and two bilingual researchers to separately translate the TREC requests into Japanese. Our main findings are as follows: (1)With ...
Exploring patent passage retrieval using nouns phrases
ECIR'13: Proceedings of the 35th European conference on Advances in Information Retrieval

This paper presents experiments which initially were carried out for the Patent Passage Retrieval track of CLEF-IP 2012. The Passage Retrieval module was implemented independently of the Document Retrieval system. In the Passage Retrieval module we make ...

Comments

Information & Contributors

Information

Published In

SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

July 2016

1296 pages

ISBN:9781450340694

DOI:10.1145/2911451

General Chairs:
Raffaele Perego
ISTI-CNR, Italy
,
Fabrizio Sebastiani
Qatar Computing Research Institute, HBKU, Qatar
,
Program Chairs:
Javed Aslam
Northeastern University, US
,
Ian Ruthven
University of Strathclyde, UK
,
Justin Zobel
University of Melbourne, Australia

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '16

Sponsor:

SIGIR

SIGIR '16: The 39th International ACM SIGIR conference on research and development in Information Retrieval

July 17 - 21, 2016

Pisa, Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
166
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dietz LHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)A Workbench for Autograding Retrieve/Generate SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657871(1963-1972)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657871
Lima de Oliveira LRomeu RMoreira VDiaz FShah CSuel TCastells PJones RSakai T(2021)REGIS: A Test Collection for Geoscientific Documents in PortugueseProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463256(2363-2368)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463256
Rücklé ASwarnkar KGurevych I(2019)Improved Cross-Lingual Question Retrieval for Community Question AnsweringThe World Wide Web Conference10.1145/3308558.3313502(3179-3186)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313502
Liu PDeng YZhu CHu H(2019)XCMRC: Evaluating Cross-Lingual Machine Reading ComprehensionNatural Language Processing and Chinese Computing10.1007/978-3-030-32233-5_43(552-564)Online publication date: 9-Oct-2019
https://dl.acm.org/doi/10.1007/978-3-030-32233-5_43
Kamateri ETsikrika TSymeonidis SVrochidis SMinker WKompatsiaris Y(2019)A Test Collection for Passage Retrieval Evaluation of Spanish Health-Related ResourcesAdvances in Information Retrieval10.1007/978-3-030-15719-7_19(148-154)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-15719-7_19

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Enhancing relevance models with adaptive passage retrieval

MT-based Japanese-Enlish cross-language IR experiments using the TREC test collections

Exploring patent passage retrieval using nouns phrases