research-article

Generating Images Instead of Retrieving Them: Relevance Feedback on Generative Adversarial Networks

Authors:

Pyry Joona, and

Tuukka RuotsaloAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

Pages 1329 - 1338

https://doi.org/10.1145/3397271.3401129

Published: 25 July 2020 Publication History

Abstract

Finding images matching a user's intention has been largely based on matching a representation of the user's information needs with an existing collection of images. For example, using an example image or a written query to express the information need and retrieving images that share similarities with the query or example image. However, such an approach is limited to retrieving only images that already exist in the underlying collection. Here, we present a methodology for generating images matching the user intention instead of retrieving them. The methodology utilizes a relevance feedback loop between a user and generative adversarial neural networks (GANs). GANs can generate novel photorealistic images which are initially not present in the underlying collection, but generated in response to user feedback. We report experiments (N=29) where participants generate images using four different domains and various search goals with textual and image targets. The results show that the generated images match the tasks and outperform images selected as baselines from a fixed image collection. Our results demonstrate that generating new information can be more useful for users than retrieving it from a collection of existing information.

References

[1]

Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky. 2014. Neural Codes for Image Retrieval. In Computer Vision -- ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 584--599.

[2]

Yang Cao, Hai Wang, Changhu Wang, Zhiwei Li, Liqing Zhang, and Lei Zhang. 2010. MindFinder: Interactive Sketch-based Image Search on Millions of Images. In Proceedings of the 18th ACM International Conference on Multimedia (Firenze, Italy) (MM '10). ACM, New York, NY, USA, 1605--1608. https://doi.org/10.1145/1873951.1874299

Digital Library

[3]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]

Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2008. Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM Comput. Surv., Vol. 40, 2, Article 5 (May 2008), 60 pages. https://doi.org/10.1145/1348246.1348248

Digital Library

[5]

Chris Donahue, Julian J. McAuley, and Miller S. Puckette. 2018. Synthesizing Audio with GANs. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. https://openreview.net/forum?id=r1RwYIJPM

[6]

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K. Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]

Abel Gonzalez-Garcia, Joost van de Weijer, and Yoshua Bengio. 2018. Image-to-image translation for cross-domain disentanglement. In Advances in Neural Information Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Curran Associates, Inc., 1287--1298. http://papers.nips.cc/paper/7404-image-to-image-translation-for-cross-domain-disentanglement.pdf

[8]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada. 2672--2680.

Digital Library

[9]

Longteng Guo, Jing Liu, Yuhang Wang, Zhonghua Luo, Wei Wen, and Hanqing Lu. 2017. Sketch-Based Image Retrieval Using Generative Adversarial Networks. In Proceedings of the 25th ACM International Conference on Multimedia (Mountain View, California, USA) (MM '17). Association for Computing Machinery, New York, NY, USA, 1267--1268. https://doi.org/10.1145/3123266.3127939

Digital Library

[10]

L. He, X. Xu, H. Lu, Y. Yang, F. Shen, and H. T. Shen 2017. Unsupervised cross-modal retrieval through adversarial learning. In 2017 IEEE International Conference on Multimedia and Expo (ICME). 1153--1158.https://doi.org/10.1109/ICME.2017.8019549

[11]

Artur Kadurin, Alexander Aliper, Andrey Kazennov, Polina Mamoshina, Quentin Vanhaelen, Kuzma Khrabrov, and Alex Zhavoronkov. 2017. The cornucopia of meaningful leads: Applying deep adversarial autoencoders for new molecule development in oncology. Oncotarget, Vol. 8, 7 (2017), 10883.

[12]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. https://openreview.net/forum?id=Hk99zCeAb

[13]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. 4401--4410.

[14]

Christoph Kofler, Martha Larson, and Alan Hanjalic. 2016. User Intent in Multimedia Search: A Survey of the State of the Art and Future Challenges. ACM Comput. Surv., Vol. 49, 2, Article 36 (Aug. 2016), 37 pages. https://doi.org/10.1145/2954930

Digital Library

[15]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

Digital Library

[16]

Wenhao Lu, Jingdong Wang, Xian-Sheng Hua, Shengjin Wang, and Shipeng Li. 2011. Contextual Image Search. In Proceedings of the 19th ACM International Conference on Multimedia (Scottsdale, Arizona, USA) (MM '11). ACM, New York, NY, USA, 513--522. https://doi.org/10.1145/2072298.2072365

Digital Library

[17]

Yongyi Lu, Shangzhe Wu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Image Generation from Sketch Constraint Using Contextual GAN. In The European Conference on Computer Vision (ECCV).

[18]

Mathias Lux, Christoph Kofler, and Oge Marques. 2010. A Classification Scheme for User Intentions in Image Search. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA '10). ACM, New York, NY, USA, 3913--3918. https://doi.org/10.1145/1753846.1754078

Digital Library

[19]

Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge university press.

[20]

Neil O'Hare, Paloma de Juan, Rossano Schifanella, Yunlong He, Dawei Yin, and Yi Chang. 2016. Leveraging User Interaction Signals for Web Image Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR '16). ACM, New York, NY, USA, 559--568. https://doi.org/10.1145/2911451.2911532

Digital Library

[21]

Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. 2016. Variational Autoencoder for Deep Learning of Images, Labels and Captions. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 2352--2360. http://papers.nips.cc/paper/6528-variational-autoencoder-for-deep-learning-of-images-labels-and-captions.pdf

[22]

Yong Rui, Thomas S. Huang, Michael Ortega-Binderberger, and Sharad Mehrotra. 1998. Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. Circuits Syst. Video Techn., Vol. 8 (1998), 644--655.

Digital Library

[23]

Tuukka Ruotsalo, Giulio Jacucci, and Samuel Kaski. 2019. Interactive faceted query suggestion for exploratory search: Whole-session effectiveness and interaction engagement. Journal of the Association for Information Science and Technology (2019).

[24]

Tuukka Ruotsalo, Giulio Jacucci, Petri Myllymäki, and Samuel Kaski. 2014. Interactive intent modeling: Information discovery beyond search. Commun. ACM, Vol. 58, 1 (2014), 86--92.

Digital Library

[25]

Tuukka Ruotsalo, Jaakko Peltonen, Manuel J. A. Eugster, Dorota Głowacka, Patrik Floréen, Petri Myllymäki, Giulio Jacucci, and Samuel Kaski. 2018. Interactive Intent Modeling for Exploratory Search. ACM Trans. Inf. Syst., Vol. 36, 4, Article 44 (Oct. 2018), 46 pages. https://doi.org/10.1145/3231593

Digital Library

[26]

Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis With Sketch and Color. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]

A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, 12 (Dec 2000), 1349--1380. https://doi.org/10.1109/34.895972

Digital Library

[28]

Bart Thomee and Michael S. Lew. 2012. Interactive search in image retrieval: a survey. International Journal of Multimedia Information Retrieval, Vol. 1, 2 (01 Jul 2012), 71--86. https://doi.org/10.1007/s13735-012-0014--4

[29]

Tung Vuong, Miamaria Saastamoinen, Giulio Jacucci, and Tuukka Ruotsalo. 2019. Understanding user behavior in naturalistic information search tasks. Journal of the Association for Information Science and Technology, Vol. 70, 11 (2019), 1248--1261.

Digital Library

[30]

Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37 (Lille, France) (ICML'15). JMLR.org, 2048--2057. http://dl.acm.org/citation.cfm?id=3045118.3045336

[31]

Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang. 2017. MidiNet: A Convolutional Generative Adversarial Network for Symbolic-Domain Music Generation. In Proceedings of the 18th International Society for Music Information Retrieval Conference, ISMIR 2017, Suzhou, China, October 23--27, 2017. 324--331.

[32]

Zheng-Jun Zha, Linjun Yang, Tao Mei, Meng Wang, Zengfu Wang, Tat-Seng Chua, and Xian-Sheng Hua. 2010. Visual Query Suggestion: Towards Capturing User Intent in Internet Image Search. ACM Trans. Multimedia Comput. Commun. Appl., Vol. 6, 3, Article 13 (Aug. 2010), 19 pages. https://doi.org/10.1145/1823746.1823747

Digital Library

[33]

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. 2017. StackGAN: Text to Photo-Realistic Image Synthesis With Stacked Generative Adversarial Networks. In The IEEE International Conference on Computer Vision (ICCV).

[34]

Xiang Sean Zhou and Thomas S. Huang. 2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, Vol. 8, 6 (01 Apr 2003), 536--544. https://doi.org/10.1007/s00530-002-0070-3

Cited By

Liu YMedlar AGłowacka D(2024)Sample, Nudge and Rank: Exploiting Interpretable GAN Controls for Exploratory SearchProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645156(582-596)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645156
Deckers NFröbe MKiesel JPandolfo GSchröder CStein BPotthast M(2023)The Infinite Index: Information Retrieval on Generative Text-To-Image ModelsProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578327(172-186)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578327
Spape MDavis KKangassalo LRavaja NSovijarvi-Spape ZRuotsalo T(2023)Brain-Computer Interface for Generating Personally Attractive ImagesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.305904314:1(637-649)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3059043
Show More Cited By

Index Terms

Generating Images Instead of Retrieving Them: Relevance Feedback on Generative Adversarial Networks
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces

Recommendations

Leveraging non-relevant images to enhance image retrieval performance
MULTIMEDIA '02: Proceedings of the tenth ACM international conference on Multimedia

Inherent subjectivity in user's perception of an image has motivated the use of relevance feedback (RF) in the image desigined output's retrieval process. RF techniques interactively determine the user's query concept, given the user's relevance ...
Read More
Learning people co-occurrence relations by using relevance feedback for retrieving group photos
ICMR '11: Proceedings of the 1st ACM International Conference on Multimedia Retrieval

This paper proposes an image retrieval method which retrieves images of a specific person from group photos. Many query-by-example methods have focused only on the visual features of the queried person. However, since socially related people such as ...
Read More
Representativeness and Diversity in Photos via Crowd-Sourced Media Analysis
Adaptive Multimedia Retrieval: Semantics, Context, and Adaptation
Abstract
In this paper we address the problem of user-adapted image retrieval. First, we provide a survey of the performance of the existing social media retrieval platforms and highlight their limitations. In this context, we propose a hybrid, two step, ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Academy of Finland

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
393
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)4

Other Metrics

View Author Metrics

Citations

Cited By

Liu YMedlar AGłowacka D(2024)Sample, Nudge and Rank: Exploiting Interpretable GAN Controls for Exploratory SearchProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645156(582-596)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645156
Deckers NFröbe MKiesel JPandolfo GSchröder CStein BPotthast M(2023)The Infinite Index: Information Retrieval on Generative Text-To-Image ModelsProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578327(172-186)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578327
Spape MDavis KKangassalo LRavaja NSovijarvi-Spape ZRuotsalo T(2023)Brain-Computer Interface for Generating Personally Attractive ImagesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.305904314:1(637-649)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3059043
Rajabi NChernik CReichlin ATaleb FVasco MGhadirzadeh ABjörkman MKragic D(2023)Mental Face Image Retrieval Based on a Closed-Loop Brain-Computer InterfaceAugmented Cognition10.1007/978-3-031-35017-7_3(26-45)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35017-7_3
Liu YMedlar AGlowacka DAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)ROGUE: A System for Exploratory Search of GANsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531675(3278-3282)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531675
Ruotsalo TWeber SGajos K(2022)Active tag recommendation for interactive entity searchInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10285659:2Online publication date: 9-May-2022
https://dl.acm.org/doi/10.1016/j.ipm.2021.102856
Parker JServati MDiller ECao SHo CLober RCohen‐Gadol A(2022)Targeting intra‐tumoral heterogeneity of human brain tumors with in vivo imaging: A roadmap for imaging genomics from multiparametric MR signalsMedical Physics10.1002/mp.1605950:4(2590-2606)Online publication date: 25-Nov-2022
https://doi.org/10.1002/mp.16059
Kropotov IMedlar AGlowacka DDemartini GZuccon GCulpepper JHuang ZTong H(2021)Exploratory Search of GANs with Contextual BanditsProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482103(3157-3161)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482103
Ahmed HKashmola M(2021)Generating digital images of skin diseases based on deep learning2021 7th International Conference on Contemporary Information Technology and Mathematics (ICCITM)10.1109/ICCITM53167.2021.9677769(179-184)Online publication date: 25-Aug-2021
https://doi.org/10.1109/ICCITM53167.2021.9677769
de la Torre-Ortiz CSpapé MKangassalo LRuotsalo TIqbal SMacLean KChevalier FMueller S(2020)Brain Relevance Feedback for Interactive Image GenerationProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415821(1060-1070)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415821

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents