research-article

Free access

A simple sentence-level extraction algorithm for comparable data

Authors:

NAACL-Short '09: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers

Pages 93 - 96

Published: 31 May 2009 Publication History

PDF eReader

Abstract

The paper presents a novel sentence pair extraction algorithm for comparable data, where a large set of candidate sentence pairs is scored directly at the sentence-level. The sentence-level extraction relies on a very efficient implementation of a simple symmetric scoring function: a computation speed-up by a factor of 30 is reported. On Spanish-English data, the extraction algorithm finds the highest scoring sentence pairs from close to 1 trillion candidate pairs without search errors. Significant improvements in BLEU are reported by including the extracted sentence pairs into the training of a phrase-based SMT (Statistical Machine Translation) system.

References

[1]

Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. CL, 19(2):263--311.

Digital Library

Google Scholar

[2]

Pascale Fung and Percy Cheung. 2004. Mining Very-Non-Parallel Corpora: Parallel Sentence and Lexicon Extraction via Bootstrapping and EM. In Proc, of EMNLP 2004, pages 57--63, Barcelona, Spain, July.

Google Scholar

[3]

Dave Graff. 2006. LDC2006T12: Spanish Gigaword Corpus First Edition. LDC.

Google Scholar

[4]

Dave Graff. 2007. LDC2007T07: English Gigaword Corpus Third Edition. LDC.

Google Scholar

[5]

Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical Phrase-Based Translation. In Proc. of HLT-NAACL'03, pages 127--133, Edmonton, Alberta, Canada, May 27 -- June 1.

Digital Library

Google Scholar

[6]

Dragos S. Munteanu and Daniel Marcu. 2005. Improving Machine Translation Performance by Exploiting Non-Parallel Corpora. CL, 31(4):477--504.

Digital Library

Google Scholar

[7]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In In Proc. of ACL'02, pages 311--318, Philadelphia, PA, July.

Digital Library

Google Scholar

[8]

Chris Quirk, Raghavendra Udupa, and Arul Menezes. 2007. Generative Models of Noisy Translations with Applications to Parallel Fragment Extraction. In Proc. of the MT Summit XI, pages 321--327, Copenhagen, Demark, September.

Google Scholar

[9]

Philip Resnik and Noah Smith. 2003. The Web as Parallel Corpus. CL, 29(3):349--380.

Digital Library

Google Scholar

[10]

Noah A. Smith. 2002. From Words to Corpora: Recognizing Translation. In Proc. of EMNLP02, pages 95--102, Philadelphia, July.

Digital Library

Google Scholar

[11]

Matthew Snover, Bonnie Dorr, and Richard Schwartz. 2008. Language and Translation Model Adaptation using Comparable Corpora. In Proc. of EMNLP08, pages 856--865, Honolulu, Hawaii, October.

Digital Library

Google Scholar

[12]

Masao Utiyama and Hitoshi Isahara. 2003. Reliable Measures for Aligning Japanese-English News Articles and Sentences. In Proc. of ACL03, pages 72--79, Sapporo, Japan, July.

Digital Library

Google Scholar

Cited By

View all

Jehl LHieber FRiezler S(2012)Twitter translation using translation-based cross-lingual retrievalProceedings of the Seventh Workshop on Statistical Machine Translation10.5555/2393015.2393074(410-421)Online publication date: 7-Jun-2012
https://dl.acm.org/doi/10.5555/2393015.2393074
Hewavitharana SVogel SZweigenbaum PRapp RSharoff S(2011)Extracting parallel phrases from comparable dataProceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web10.5555/2024236.2024248(61-68)Online publication date: 24-Jun-2011
https://dl.acm.org/doi/10.5555/2024236.2024248
Gahbiche-Braham SBonneau-Maynard HYvon FZweigenbaum PRapp RSharoff S(2011)Two ways to use a noisy parallel news corpus for improving statistical machine translationProceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web10.5555/2024236.2024246(44-51)Online publication date: 24-Jun-2011
https://dl.acm.org/doi/10.5555/2024236.2024246
Show More Cited By

Index Terms

A simple sentence-level extraction algorithm for comparable data
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
  2. Modeling and simulation
    1. Model development and analysis
      1. Model verification and validation
      2. Modeling methodologies
2. Information systems
  1. Information retrieval
    1. Information retrieval query processing

Recommendations

A Hybrid of Sentence-Level Approach and Fragment-Level Approach of Parallel Text Extraction from Comparable Text
Abstract
Parallel texts are essential resources in linguistics, natural language processing, and multilingual information retrieval. Many studies attempt to extract parallel text from existing resources, particularly from comparable texts. The approaches ...
The Cognitive Interactionist Approach of Sentence Parsing with Simple Recurrent Networks
IITA '08: Proceedings of the 2008 Second International Symposium on Intelligent Information Technology Application - Volume 01

Sentence parsing has a long research history in the fields of machine learning and natural language processing. The state-of-the-art techniques for tackling this task are mostly based on statistical language learning. How human parsing sentences is also ...
Parallel sentence generation from comparable corpora for improved SMT

A parallel corpus is an essential resource for statistical machine translation (SMT) but is often not available in the required amounts for all domains and languages. An approach is presented here which aims at producing parallel corpora from available ...

Comments

Information & Contributors

Information

Published In

NAACL-Short '09: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers

May 2009

317 pages

Publisher

Association for Computational Linguistics

United States

Publication History

Published: 31 May 2009

Qualifiers

Research-article

Acceptance Rates

Overall Acceptance Rate 21 of 29 submissions, 72%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
299
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)5

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Jehl LHieber FRiezler S(2012)Twitter translation using translation-based cross-lingual retrievalProceedings of the Seventh Workshop on Statistical Machine Translation10.5555/2393015.2393074(410-421)Online publication date: 7-Jun-2012
https://dl.acm.org/doi/10.5555/2393015.2393074
Hewavitharana SVogel SZweigenbaum PRapp RSharoff S(2011)Extracting parallel phrases from comparable dataProceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web10.5555/2024236.2024248(61-68)Online publication date: 24-Jun-2011
https://dl.acm.org/doi/10.5555/2024236.2024248
Gahbiche-Braham SBonneau-Maynard HYvon FZweigenbaum PRapp RSharoff S(2011)Two ways to use a noisy parallel news corpus for improving statistical machine translationProceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web10.5555/2024236.2024246(44-51)Online publication date: 24-Jun-2011
https://dl.acm.org/doi/10.5555/2024236.2024246
Smith JQuirk CToutanova KKaplan R(2010)Extracting parallel sentences from comparable corpora using document level alignmentHuman Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics10.5555/1857999.1858062(403-411)Online publication date: 2-Jun-2010
https://dl.acm.org/doi/10.5555/1857999.1858062
Tillmann C(2009)A beam-search extraction algorithm for comparable dataProceedings of the ACL-IJCNLP 2009 Conference Short Papers10.5555/1667583.1667653(225-228)Online publication date: 4-Aug-2009
https://dl.acm.org/doi/10.5555/1667583.1667653

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

A Hybrid of Sentence-Level Approach and Fragment-Level Approach of Parallel Text Extraction from Comparable Text

The Cognitive Interactionist Approach of Sentence Parsing with Simple Recurrent Networks

Parallel sentence generation from comparable corpora for improved SMT

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations