article

Time-based tags for fiction movies: comparing experts to novices using a video labeling game

Authors:

Liliana Melgar Estrada,

Michiel Hildebrand,

Victor de Boer,

Jacco van OssenbruggenAuthors Info & Claims

Journal of the Association for Information Science and Technology, Volume 68, Issue 2

Pages 348 - 364

https://doi.org/10.1002/asi.23656

Published: 01 February 2017 Publication History

Abstract

The cultural heritage sector has embraced social tagging as a way to increase both access to online content and to engage users with their digital collections. In this article, we build on two current lines of research. a We use Waisda?, an existing labeling game, to add time-based annotations to content. b In this context, we investigate the role of experts in human-based computation nichesourcing. We report on a small-scale experiment in which we applied Waisda? to content from film archives. We study the differences in the type of time-based tags between experts and novices for film clips in a crowdsourcing setting. The findings show high similarity in the number and type of tags mostly factual. In the less frequent tags, however, experts used more domain-specific terms. We conclude that competitive games are not suited to elicit real expert-level descriptions. We also confirm that providing guidelines, based on conceptual frameworks that are more suited to moving images in a time-based fashion, could result in increasing the quality of the tags, thus allowing for creating more tag-based innovative services for online audiovisual heritage.

References

[1]

Ådland, M.K., &Lykke, M. 2012. Social tagging in support of cancer patients' information interaction. In Social information research Vol. Volume 5, pp. pp.101-128. UK: Emerald Group Publishing.

[2]

Ahn, L.von, &Dabbish, L. 2008. Designing games with a purpose. Communications of the ACM, Volume 51 Issue 8, pp.58-67.

Digital Library

[3]

Baca, M. 2002. Introduction to art image access: Issues, tools, standards, and strategies. Los Angeles, CA: Getty Research Institute.

Digital Library

[4]

Bálint, K., &Kovács, A.B. 2012. Focalization and attachment. Studying the interaction effect of narrative and psychological factors in film viewers' emotional responses. Pszichológia, Volume 32 Issue 3, pp.271-291.

[5]

Ballan, L., Bertini, M., Del Bimbo, A., Meoni, M., &Serra, G. 2010. Tag suggestion and localization in user-generated videos based on social knowledge. In Proceedings of Second ACM SIGMM Workshop on Social Media pp. 3-8. New York, NY: ACM.

Digital Library

[6]

Ballan, L., Bertini, M., Del Bimbo, A., &Serra, G. 2011. Enriching and localizing semantic tags in Internet videos. In Proceedings of the 19th ACM International Conference on Multimedia pp. 1541-1544. New York, NY: ACM.

Digital Library

[7]

Bar-Ilan, J., Shoham, S., Idan, A., Miller, Y., &Shachak, A. 2008. Structured versus unstructured tagging: A case study. Online Information Review, Volume 32 Issue 5, pp.635-647.

[8]

Bertini, M., Del Bimbo, A., Ferracani, A., Gelli, F., Maddaluno, D., &Pezzatini, D. 2013a. A novel framework for collaborative video recommendation, interest discovery and friendship suggestion based on semantic profiling. In Proceedings of the 21st ACM International Conference on Multimedia pp. 451-452. New York, NY: ACM.

Digital Library

[9]

Bertini, M., Del Bimbo, A., Ferracani, A., Gelli, F., Maddaluno, D., &Pezzatini, D. 2013b. Socially-aware video recommendation using users' profiles and crowdsourced annotations. In Proceedings of the 2nd International Workshop on Socially-aware Multimedia pp. 13-18. New York, NY: ACM.

Digital Library

[10]

De Boer, V., Hildebrand, M., Aroyo, L., Leenheer, P.D., Dijkshoorn, C., Tesfa, B., &Schreiber, G. 2012. Nichesourcing: Harnessing the power of crowds of experts. In A.ten Teije, J.Völker, S.Handschuh, H.Stuckenschmidt, M.d'Acquin, A.Nikolov, ' N.Hernandez Eds., Ekaw'12: Proceedings of the 18th International Conference on Knowledge Engineering and Knowledge Management pp. pp.16-20. Berlin, Heidelberg: Springer-Verlag. Retrieved from

Digital Library

[11]

Bordwell, D., &Thompson, K. 2003. Film art: An introduction 7th ed. Boston: Mcgraw-Hill College.

[12]

Burford, B., Briggs, P., &Eakins, J.P. 2003. A taxonomy of the image: On the classification of content for image retrieval. Visual Communication, Volume 2 Issue 2, pp.123-161.

[13]

Darvish, S., &Chin, A. 2010. Dealing with the video tidal wave: The relevance of expertise for video tagging. In Proceedings of the 21st ACM Conference on Hypertext and Hypermedia pp. 289-290. New York, NY: ACM.

Digital Library

[14]

Eakins, J.P., Briggs, P., &Burford, B. 2004. Image retrieval interfaces: A user perspective. In P.G.B.Enser, Y.Kompatsiaris, N.E.O'Connor, A.F.Smeaton, &A.W.M.Smeulders Eds., Image and video retrieval pp. pp.628-637. Berlin, Heildelberg: Springer.

[15]

Enser, P.G.B. 2000. Visual image retrieval: Seeking the alliance of concept-based and content-based paradigms. Journal of Information Science, Volume 26 Issue 4, pp.199-210.

[16]

Fossati, G. 2009. From grain to pixel: The archival life of film in transition. Amsterdam: Amsterdam University Press.

[17]

Freiburg, B., Kamps, J., &Snoek, C.G.M. 2011. Crowdsourcing visual detectors for video search pp. pp.913-916. New York, NY: ACM.

Digital Library

[18]

Fu, W.-T., Kannampallil, T., Kang, R., &He, J. 2010. Semantic imitation in social tagging. ACM Transactions on Computer-Human Interaction, Volume 17 Issue 3, 12:1-12:37.

Digital Library

[19]

Gedikli, F., &Jannach, D. 2013. Improving recommendation accuracy based on item-specific tag preferences. ACM Transaction on Intelligent Systems and Technology, Volume 4 Issue 1, 11:1-11:19.

Digital Library

[20]

Geisler, G., Willard, G., &Whitworth, E. 2010. Crowdsourcing the indexing of film and television media pp. 82:1-82:10. Silver Springs, MD: American Society for Information Science. Retrieved from "http://dl.acm.org/citation.cfm?id=1920331.1920448"

Digital Library

[21]

Gibbon, D.C., Liu, Z., Basso, A., &Shahraray, B. 2013. Automated content metadata extraction services based on MPEG standards. Computer Journal, Volume 56 Issue 5, pp.628-645.

Digital Library

[22]

Gligorov, R., Hildebrand, M., <familyNamePrefix>van</familyNamePrefix>Ossenbruggen, J., Aroyo, L., &Schreiber, G. 2013. An evaluation of labelling-game data for video retrieval. In P.Serdyukov, P.Braslavski, S.O.Kuznetsov, J.Kamps, S.Rüger, E.Agichtein, ' E.Yilmaz Eds., Advances in information retrieval pp. pp.50-61. Berlin, Heildelberg: Springer.

Digital Library

[23]

Gligorov, R., Hildebrand, M., <familyNamePrefix>van</familyNamePrefix>Ossenbruggen, J., Schreiber, G., &Aroyo, L. 2011. On the role of user-generated metadata in audio visual collections pp. pp.145-152. New York, NY: ACM.

Digital Library

[24]

Goh, D.H.-L., Ang, R.P., Lee, C.S., &Chua, A.Y.K. 2011. Fight or unite: Investigating game genres for image tagging. Journal of the American Society for Information Science and Technology, Volume 62 Issue 7, pp.1311-1324.

Digital Library

[25]

Goh, D.H.-L., &Lee, C.S. 2011. Perceptions, quality and motivational needs in image tagging human computation games. Journal of Information Science, Volume 37 Issue 5, pp.515-531.

Digital Library

[26]

Golbeck, J., Koepfler, J., &Emmerling, B. 2011. An experimental study of social tagging behavior and image content. Journal of the American Society for Information Science and Technology, Volume 62 Issue 9, pp.1750-1760.

Digital Library

[27]

Good, B.M., Tennis, J.T., &Wilkinson, M.D. 2009. Social tagging in the life sciences: Characterizing a new metadata resource for bioinformatics. BMC Bioinformatics, Volume 10, pp.313-313.

[28]

Greisdorf, H., &O'Connor, B., 2002. Modelling what users see when they look at images: a cognitive viewpoint. Journal of Documentation, Volume 58 Issue 1, pp.6-29.

[29]

Guy, M., &Tonkin, E. 2006. Folksonomies: Tidying up tags? D-Lib Magazine, Volume 12 Issue 1. Retrieved from "http://webdoc.sub.gwdg.de/edoc/aw/d-lib/dlib/january06/guy/01guy.html"

[30]

Halpin, H., Robu, V., &Shepherd, H. 2007. The complex dynamics of collaborative tagging. In Proceedings of the 16th international conference on World Wide Web pp. 211-220. New York, NY: ACM.

Digital Library

[31]

Hildebrand, M., Brinkerink, M., Gligorov, R., <familyNamePrefix>van</familyNamePrefix>Steenbergen, M., Huijkman, J., &Oomen, J. 2013. Waisda? Video Labeling Game. Presented at the ACM Multimedia, Barcelona.

Digital Library

[32]

Hollink, L. 2006. Semantic annotation for retrieval of visual resources Doctoral dissertation. Vrije Universiteit, Amsterdam. Retrieved from "http://hdl.handle.net/1871/10846"

[33]

Hollink, L., Schreiber, A.T., Wielinga, B.J., &Worring, M. 2004. Classification of user image descriptions. International Journal of Human-Computer Studies, Volume 61 Issue 5, pp.601-626.

Digital Library

[34]

Huang, C., Fu, T., &Chen, H. 2010. Text-based video content classification for online video-sharing sites. Journal of the American Society for Information Science and Technology, Volume 61 Issue 5, pp.891-906.

Digital Library

[35]

Images for the Future. 2009. Waisda? Video labeling game: Evaluation report. Retrieved from "http://research.imagesforthefuture.org/index.php/waisda-video-labeling-game-evaluation-report"/

[36]

Ingwersen, P. 1992. Information retrieval interaction. London: Taylor Graham.

Digital Library

[37]

Inskip, C., MacFarlane, A., &Rafferty, P. 2008. Content or context?: Searching for musical meaning in task-based interactive information retrieval. In Proceedings of the Second International Symposium on Information Interaction in Context pp. pp.72-74. New York, NY: ACM.

Digital Library

[38]

Kang, R., &Fu, W.-T. 2010. Exploratory information search by domain experts and novices. In Proceedings of the 15th International Conference on Intelligent User Interfaces pp. pp.329-332. New York, NY: ACM.

Digital Library

[39]

Kim, H.H., &Kim, Y.H. 2010. Toward a conceptual framework of key-frame extraction and storyboard display for video summarization. Journal of the American Society for Information Science and Technology, Volume 61 Issue 5, pp.927-939.

Digital Library

[40]

Klavans, J.L., LaPlante, R., &Golbeck, J. 2013. Subject matter categorization of tags applied to digital images from art museums. Journal of the American Society for Information Science and Technology, 65, 3-12.

[41]

Layne, S.S. 1986. Analyzing the subject of a picture: A theoretical approach. Cataloging & Classification Quarterly, Volume 6 Issue 3, pp.39-62.

[42]

Lee, C.S., Goh, D.H.-L., Razikin, K., &Chua, A.Y.K. 2009. Tagging, sharing and the influence of personal experience. Journal of Digital Information, Volume 10 Issue 1. pp.1-15. Retrieved from "https://journals.tdl.org/jodi/index.php/jodi/article/view/275"

[43]

Li, G., Wang, M., Zheng, Y.-T., Li, H., Zha, Z.-J., &Chua, T.-S. 2011. ShotTagger: Tag location for Internet videos. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval pp. 37:1-37:8. New York, NY: ACM.

Digital Library

[44]

Lu, C., Park, J., &Hu, X. 2010. User tags versus expert-assigned subject terms: A comparison of LibraryThing tags and library of congress subject headings. Journal of Information Science, Volume 36 Issue 6, pp.763-779.

Digital Library

[45]

Madrigal, A.C. 2014, January. How Netflix Reverse Engineered Hollywood. The Atlantic. Retrieved from "http://www.theatlantic.com/technology/archive/2014/01/how-netflix-reverse-engineered-hollywood/282679/3"/

[46]

Matusiak, K.K. 2006. Towards user-centered indexing in digital image collections. OCLC Systems & Services, Volume 22 Issue 4, pp.283-298.

[47]

Melenhorst, M., Grootveld, M., <familyNamePrefix>van</familyNamePrefix>Setten, M., &Veenstra, M. 2008. Tag-based information retrieval of video content pp. pp.31-40. New York, NY: ACM.

Digital Library

[48]

Oomen, J., &Aroyo, L. 2011. Crowdsourcing in the cultural heritage domain: Opportunities and challenges pp. pp.138-149. New York, NY: ACM.

Digital Library

[49]

Oomen, J., Gligorov, R., &Hildebrand, M. 2014. Waisda?: making videos findable through crowdsourced annotations. In M.Ridge Ed., Crowdsourcing our Cultural Heritage pp. pp.161-184. Ashgate Publishing, Ltd.

[50]

Panofsky, E. 1939. Studies in iconology: Humanistic themes in the art of the Renaissance 1st Icon ed., 4th print. New York, NY: Harper & Row.

[51]

Peters, I. 2009. Folksonomies. Indexing and Retrieval in Web 2.0 1st ed. Berlin: De Gruyter.

Digital Library

[52]

Rudkin, D. 2007. Vampyr. London: University of California Press.

[53]

Sen, S., Lam, S.K., Rashid, A.M., Cosley, D., Frankowski, D., Osterhouse, J., ' Riedl, J. 2006. Tagging, communities, vocabulary, evolution. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work pp. pp.181-190. New York, NY: ACM.

Digital Library

[54]

Skov, M., &Lykke, M. 2012. Unlocking radio broadcasts: User needs in sound retrieval pp. pp.298-301. New York, NY: ACM.

Digital Library

[55]

Smith, G. 2007. Tagging: People-powered metadata for the social web. Berkeley, CA: New Riders Press.

Digital Library

[56]

Springer, M., Dulabahn, B., Michel, P., Natanson, B., Reser, D., Woodward, D., &Zinkham, H. 2008. For the common good: The Library of Congress Flickr pilot project Report. DC: Government of the United States; Library of Congress. Retrieved from "http://www.egov.vic.gov.au/focus-on-countries/north-and-south-america-and-the-caribbean/united-states/government-initiatives-united-states/culture-sport-and-recreation-united-states/libraries-united-states/for-the-common-good-the-library-of-congress-flickr-pilot-project-in-pdf-format-1333kb-.html"

[57]

Stock, W. G. 2010. Concepts and semantic relations in information science. Journal of the American Society for Information Science and Technology, Volume 61 Issue 10, pp.1951-1969.

Digital Library

[58]

Thøgersen, R. 2013. Data quality in an output-agreement game: A comparison between game-generated tags and professional descriptors. In P.Antunes, M.A.Gerosa, A.Sylvester, J.Vassileva, &G.-J.<familyNamePrefix>de</familyNamePrefix>Vreede Eds., Collaboration and technology pp. pp.126-142. Berlin, Heildelberg: Springer.

Digital Library

[59]

Thom-Santelli, J., Cosley, D., &Gay, G. 2010. What do you know?: Experts, novices and territoriality in collaborative systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems pp. pp.1685-1694. New York, NY: ACM.

Digital Library

[60]

Tirilly, P., Mu, X., Huang, C., Xie, I., Jeong, W., &Zhang, J. 2012. On the consistency and features of image similarity pp. pp.164-173. New York, NY: ACM.

Digital Library

[61]

Trant, J. 2009a. Tagging, folksonomy and art museums: Early experiments and ongoing research. Journal of Digital Information, Volume 10 Issue 1. Retrieved from "http://journals.tdl.org/jodi/article/viewArticle/270"

[62]

Trant, J. 2009b. Tagging, Folksonomy and Art Museums: Results of steve.museum's research. Retrieved from "http://www.museumsandtheweb.com/blog/jtrant/stevemuseum_research_report_available_tagging_fo.html"

[63]

Troncy, R., Huet, B., &Schenk, S. 2011. Multimedia semantics: Metadata, analysis and interaction. United Kingdom: John Wiley & Sons.

[64]

Tsai, L.-C., Hwang, S.-L., &Tang, K.-H. 2011. Analysis of keyword-based tagging behaviors of experts and novices. Online Information Review, Volume 35 Issue 2, pp.272-290.

[65]

Turner, J.M. 2009. Moving image indexing. In Encyclopedia of library and information sciences 3rd ed., pp. pp.3671-3681. New York: Taylor & Francis.

[66]

Turner, J.M. 2010. From ABC to http: The Effervescent evolution of indexing for audiovisual materials. Cataloging & Classification Quarterly, Volume 48 Issue 1, pp.83-93.

[67]

Von Ahn, L., &Dabbish, L. 2008. Designing games with a purpose. Communications of the ACM, Volume 51 Issue 8, pp.58-67.

Digital Library

[68]

Wang, M., Ni, B., Hua, X.-S., &Chua, T.-S. 2012. Assistive tagging: A survey of multimedia tagging with human-computer joint exploration. ACM Computing Surveys, Volume 44 Issue 4, 25:1-25:24.

Digital Library

[69]

Westman, S. 2009. Image Users' Needs and Searching Behaviour. In A.Göker &J.Davies Eds., Information retrieval: Searching in the 21st century; human information retrieval pp. pp.63-83. Chichester, U.K .: Wiley. Retrieved from

[70]

Wilkie, C. 1999. Managing film and video collections. London: Aslib and Information Management International.

[71]

Winget, M. 2009. Describing art: An alternative approach to subject access and interpretation. Journal of Documentation, Volume 65 Issue 6, pp.958-976.

[72]

Yeh, M.-C., &Wu, W.-P. 2014. Clustering faces in movies using an automatically constructed social network. IEEE MultiMedia, Volume 21 Issue 2, pp.22-31.

[73]

Zollers, A. 2007. Emerging motivations for tagging: Expression, performance, and activism. In Tagging and Metadata for Social Information Organization Workshop. Banff, Canada: WWW07. Retrieved from "http://www2007.org/workshops/paper_55.pdf"

Cited By

Fan SBarlas PChristoforou EOtterbacher JSadiq SDemartini GBaeza-Yates RWeller KPortela MSeneviratne OWeber IYasseri TBon ASrinivas SIbáñez L(2022)Socio-Economic Diversity in Human AnnotationsProceedings of the 14th ACM Web Science Conference 202210.1145/3501247.3531588(98-109)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1145/3501247.3531588

Recommendations

Enriching and localizing semantic tags in internet videos
MM '11: Proceedings of the 19th ACM international conference on Multimedia

Tagging of multimedia content is becoming more and more widespread as web 2.0 sites, like Flickr and Facebook for images, YouTube and Vimeo for videos, have popularized tagging functionalities among their users. These user-generated tags are used to ...
Tags weighting based on user profile
AMT'11: Proceedings of the 7th international conference on Active media technology

The 'Collaborative Tagging' is gaining popularity on Web 2.0, this new generation of Web which makes user reader/writer. The 'Tagging' is a mean for users to express themselves freely through additions of label called 'Tags' to shared resources. One of ...
Tagging tags
MM '10: Proceedings of the 18th ACM international conference on Multimedia

Social image sharing websites like Flickr have successfully motivated users around the world to annotate images with tags, which greatly facilitate search and organization of social image content. However, these manually-input tags are far from a ...

Comments

Information & Contributors

Information

Published In

cover image Journal of the Association for Information Science and Technology

Journal of the Association for Information Science and Technology Volume 68, Issue 2

February 2017

269 pages

ISSN:2330-1635

EISSN:2330-1643

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 February 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fan SBarlas PChristoforou EOtterbacher JSadiq SDemartini GBaeza-Yates RWeller KPortela MSeneviratne OWeber IYasseri TBon ASrinivas SIbáñez L(2022)Socio-Economic Diversity in Human AnnotationsProceedings of the 14th ACM Web Science Conference 202210.1145/3501247.3531588(98-109)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1145/3501247.3531588

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents