poster

Cross-media entity recognition in nearly parallel visual and textual documents

Authors:

Koen Deschacht,

Marie-Francine Moens, and

Wouter RobeynsAuthors Info & Claims

RIAO '07: Large Scale Semantic Access to Content (Text, Image, Video, and Sound)

May 2007

Pages 133 - 144

Published: 30 May 2007 Publication History

Abstract

We present a novel approach to automatically annotate images solely using associated text. We detect and classify all entities (persons and objects) in the text after which we determine the salience (the importance of an entity in a text) and visualness (the extent to which an entity can be perceived visually) of these entities. We combine these measures to compute the probability that an entity is present in the image. The suitability of our approach was successfully tested on 900 image-text pairs of Yahoo! News.

References

[1]

A. Amir, J. Argillander, M. Campbell, A. Haubold, G. Iyengar, S. Ebadollahi, F. Kang, M. R. Naphade, A. Natsev, J. R. Smith, J. Tešió, and T. Volkmer. IBM Research TRECVID-2005 Video Retrieval System. In Proceedings of TRECVID 2005, Gaithersburg, MD, 2005.

[2]

S. Ayache, G. M. Qunot, J. Gensel, and S. Satoh. CLIPS-LRS-NII Experiments at TRECVID 2005. In Proceedings of TRECVID 2005, Gaithersburg, MD, 2005.

[3]

K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan. Matching Words and Pictures. Journal of Machine Learning Research, 3(6):1107--1135, 2003.

Digital Library

[4]

T. L. Berg, A. C. Berg, J. Edwards, and D. A. Forsyth. Who's in the Picture? In Proceedings of the 18th Annual Conference on Neural Information Processing Systems, pages 137--144, 2004.

[5]

E. Charniak. A Maximum-Entropy-Inspired Parser. In Proceedings of the First Conference on North American chapter of the Association for Computational Linguistics, pages 132--139. Morgan Kaufmann Publishers Inc. San Francisco, CA, USA, 2000.

Digital Library

[6]

K. Deschacht and M.-F. Moens. Efficient Hierarchical Entity Classification Using Conditional Random Fields. In Proceedings of the 2nd Workshop on Ontology Learning and Population, pages 33--40, Sydney, July 2006.

[7]

C. Fellbaum. WordNet: An Electronic Lexical Database. The MIT Press, 1998.

[8]

B. Hayes. The Web of Words. American Scientist, 87(2):108--112, March-April 1999.

[9]

J. Kamps and M. Marx. Words with Attitude. In Proceedings of the 1st International Conference on Global WordNet, pages 332--341, Mysore, IN, 2002.

[10]

S. Landes, C. Leacock, and R. I. Tengi. Building Semantic Concordances. In C. Fellbaum, editor, WordNet: An Electronic Lexical Database. The MIT Press, 1998.

[11]

D. Lin. An Information-Theoretic Definition of Similarity. In Proceedings of the 15th International Conf. on Machine Learning, 1998.

Digital Library

[12]

A. Mikheev. Automatic Rule Induction for Unknown-Word Guessing. Computational Linguistics, 23(3):405--423, 1997.

Digital Library

[13]

M.-F. Moens. Using Patterns of Thematic Progression for Building a Table of Content of a Text. Journal of Natural Language Engineering, 12(3):1--28, 2006.

Digital Library

[14]

M.-F. Moens, R. Angheluta, and J. Dumortier. Generic Technologies for Single- and Multi-Document Summarization. Information Processing and Management, 41(3):569--586, 2005.

Digital Library

[15]

M.-F. Moens, P. Jeuniaux, R. Angheluta, and R. Mitra. Measuring Aboutness of an Entity in a Text. In Proceedings of HLT-NAACL 2006 TextGraphs: Graph-based Algorithms for Natural Language Processing, East Stroudsburg, 2006. ACL.

Digital Library

[16]

Y. Mori, H. Takahashi, and R. Oka. Automatic Word Assignment to Images Based on Image Division and Vector Quantization. In RIAO-2000 Content-Based Multimedia Information Access, Paris, April 12--14 2000.

[17]

T. Pedersen, S. Patwardhan, and J. Michelizzi. WordNet::Similarity - Measuring the Relatedness of Concepts. In The Proceedings of Fifth Annual Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL-04), Boston, May 2004.

Digital Library

[18]

S. Satoh, Y. Nakamura, and T. Kanade. Name-It: Naming and Detecting Faces in News Videos. IEEE MultiMedia, 6(1):22--35, January-March 1999.

Digital Library

[19]

P. Viola and M. Jones. Rapid Object Detection Using a Boosted Cascade of Simple Features. Proc. CVPR, 1:511--518, 2001.

[20]

T. Westerveld. Image Retrieval: Content versus Context. In Proceedings of the RIAO 2000 conference: Content-Based Multimedia Information Access, pages 276--284, April 2000. ISBN 2-905450-07-X.

[21]

T. Westerveld, J. C. van Gemert, R. Cornacchia, D. Hiemstra, and A. de Vries. An Integrated Approach to Text and Image Retrieval. In Proceedings of TRECVID 2005, Gaithersburg, MD, 2005.

Cited By

Rizk YJomaa HAwad MCastillo CHung CPapadopoulos G(2019)A computationally efficient multi-modal classification approach of disaster-related Twitter imagesProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3297481(2050-2059)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1145/3297280.3297481
De Belder JMoens M(2012)A dataset for the evaluation of lexical simplificationProceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II10.1007/978-3-642-28601-8_36(426-437)Online publication date: 11-Mar-2012
https://dl.acm.org/doi/10.1007/978-3-642-28601-8_36

Index Terms

Cross-media entity recognition in nearly parallel visual and textual documents

Recommendations

Using Wikipedia for cross-language named entity recognition
MSM/MUSE/SenseML'14: Proceedings of the 5th and 1st International Conference on Big Data Analytics in the Social and Ubiquitous Context - 5th International Workshop on Modeling Social Media, 5th International Workshop on Mining Ubiquitous and Social Environments and First International Workshop on Machine Learning for Urban Sensor Data

Named entity recognition and classification (NERC) is fundamental for natural language processing tasks such as information extraction, question answering, and topic detection. State-of-the-art NERC systems are based on supervised machine learning and ...
Read More
A joint named entity recognition and entity linking system
HYBRID '12: Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data

We present a joint system for named entity recognition (NER) and entity linking (EL), allowing for named entities mentions extracted from textual data to be matched to uniquely identifiable entities. Our approach relies on combined NER modules which ...
Read More
Named entity recognition in Wikipedia
People's Web '09: Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources

Named entity recognition (NER) is used in many domains beyond the newswire text that comprises current gold-standard corpora. Recent work has used Wikipedia's link structure to automatically generate near gold-standard annotations. Until now, these ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RIAO '07: Large Scale Semantic Access to Content (Text, Image, Video, and Sound)

May 2007

817 pages

Sponsors

CID (France): Le Centre de Hautes Etudes Internationales D'Informatique Documentaire

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

LE CENTRE DE HAUTES ETUDES INTERNATIONALES D'INFORMATIQUE DOCUMENTAIRE

Paris, France

Publication History

Published: 30 May 2007

Check for updates

Qualifiers

Poster

Conference

RIAO07

Sponsor:

CID (France)

RIAO07: Large Scale Semantic Access to Content

May 30 - June 1, 2007

Pennsylvania, Pittsburgh

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
52
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Rizk YJomaa HAwad MCastillo CHung CPapadopoulos G(2019)A computationally efficient multi-modal classification approach of disaster-related Twitter imagesProceedings of the 34th ACM/SIGAPP Symposium on Applied Computing10.1145/3297280.3297481(2050-2059)Online publication date: 8-Apr-2019
https://dl.acm.org/doi/10.1145/3297280.3297481
De Belder JMoens M(2012)A dataset for the evaluation of lexical simplificationProceedings of the 13th international conference on Computational Linguistics and Intelligent Text Processing - Volume Part II10.1007/978-3-642-28601-8_36(426-437)Online publication date: 11-Mar-2012
https://dl.acm.org/doi/10.1007/978-3-642-28601-8_36

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents