research-article

Brain Relevance Feedback for Interactive Image Generation

Authors:

Carlos de la Torre-Ortiz,

Michiel M. Spapé,

Lauri Kangassalo,

Tuukka RuotsaloAuthors Info & Claims

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Pages 1060 - 1070

https://doi.org/10.1145/3379337.3415821

Published: 20 October 2020 Publication History

Abstract

Brain-computer interfaces (BCIs) are increasingly used to perform simple operations such as a moving a cursor, but have remained of limited use for more complex tasks. In our new approach to BCI, we use brain relevance feedback to control a generative adversarial network (GAN). We obtained EEG data from 31 participants who viewed face images while concentrating on particular facial features. Following, an EEG relevance classifier was trained and propagated as feedback on the latent image representation provided by the GAN. Estimates for individual vectors matching the relevant criteria were iteratively updated to optimize an image generation process towards mental targets. A double-blind evaluation showed high performance (86.26% accuracy) against random feedback (18.71%), and not significantly lower than explicit feedback (93.30%). Furthermore, we show the feasibility of the method with simultaneous task targets demonstrating BCI operation beyond individual task constraints. Thus, brain relevance feedback can validly control a generative model, overcoming a critical limitation of current BCI approaches.

Supplementary Material

VTT File (3379337.3415821.vtt)

Download
3.46 KB

ZIP File (ufp2231aux.zip)

Brain Relevance Feedback for Interactive Image Generation -- Supplementary

Download
33.30 MB

MP4 File (3379337.3415821.mp4)

Presentation Video

Download
27.73 MB

References

[1]

Emanuel Donchin and Michael GH Coles. 1988. Is the P300 component a manifestation of context updating? Behavioral and brain sciences 11, 03 (1988), 357--374.

[2]

Emanuel Donchin, Kevin M Spencer, and Ranjith Wijesinghe. 2000. The mental prosthesis: assessing the speed of a P300-based brain-computer interface. IEEE transactions on rehabilitation engineering 8, 2 (2000), 174--179.

[3]

Manuel JA Eugster, Tuukka Ruotsalo, Michiel M Spapé, Ilkka Kosunen, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2014. Predicting term-relevance from brain signals. In Proc. SIGIR. ACM, 425--434.

Digital Library

[4]

Manuel J. A. Eugster, Tuukka Ruotsalo, Michiel M. Spapé, Oswald Barral, Niklas Ravaja, Giulio Jacucci, and Samuel Kaski. 2016. Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals. Scientifc Reports 6 (Dec. 2016), 38580.

[5]

Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K. Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back. In Proc. CVPR.

[6]

Lawrence Ashley Farwell and Emanuel Donchin. 1988. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and clinical Neurophysiology 70, 6 (1988), 510--523.

[7]

P.I. Good. 2000. Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses (2nd ed.). Springer.

[8]

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Proc. NIPS. 2672--2680.

Digital Library

[9]

Longteng Guo, Jing Liu, Yuhang Wang, Zhonghua Luo, Wei Wen, and Hanqing Lu. 2017. Sketch-Based Image Retrieval Using Generative Adversarial Networks. In Proc. MM. Association for Computing Machinery, New York, NY, USA, 1267--1268.

Digital Library

[10]

Giulio Jacucci, Oswald Barral, Pedram Daee, Markus Wenzel, Baris Serim, Tuukka Ruotsalo, Patrik Pluchino, Jonathan Freeman, Luciano Gamberini, Samuel Kaski, and others. 2019. Integrating neurophysiologic relevance feedback in intent modeling for information retrieval. Journal of the Association for Information Science and Technology 70, 9 (2019), 917--930.

Digital Library

[11]

Lauri Kangassalo, Michiel Spapé, Giulio Jacucci, and Tuukka Ruotsalo. 2019. Why Do Users Issue Good Queries? Neural Correlates of Term Specifcity. In Proc. SIGIR. ACM, New York, NY, USA, 375--384.

[12]

Lauri Kangassalo, Michiel Spapé, Niklas Ravaja, and Tuukka Ruotsalo. 2020. Information gain modulates brain activity evoked by reading. Scientifc reports 10, 1 (2020), 1--10.

[13]

Lauri Kangassalo, Michiel M. Spapé, and Tuukka Ruotsalo. 2020. Neuroadaptive modelling for generating images matching perceptual categories. Scientifc Reports (2020), to appear.

[14]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In Proc. ICLR.

[15]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. In Proc. CVPR. 4401--4410.

[16]

Isaak Kavasidis, Simone Palazzo, Concetto Spampinato, Daniela Giordano, and Mubarak Shah. 2017. Brain2Image: Converting Brain Signals into Images. In Proc. MM. ACM, New York, NY, USA, 1809--1817.

Digital Library

[17]

Boris Kotchoubey. 2006. Event-related potentials, cognition, and behavior: a biological approach. 30, 1 (2006), 42--65.

[18]

Ren Li, Jared S Johansen, Hamad Ahmed, Thomas V Ilyevsky, Ronnie B Wilbur, Hari M Bharadwaj, and Jeffrey Mark Siskind. 2018. Training on the test set? An analysis of Spampinato et al.[arXiv: 1609.00344]. arXiv preprint arXiv:1812.07697 (2018).

[19]

Z. Liu, P. Luo, X. Wang, and X. Tang. 2015. Deep Learning Face Attributes in the Wild. In Proc. ICCV. 3730--3738.

[20]

Yongyi Lu, Shangzhe Wu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Image Generation from Sketch Constraint Using Contextual GAN. In Proc. ECCV.

[21]

Markus Ojala and Gemma C. Garriga. 2009. Permutation Tests for Studying Classifer Performance. In Proc. Data Mining. IEEE, Miami Beach, FL, USA, 908--913.

[22]

Terence W Picton. 1992. The P300 wave of the human event-related potential. Journal of clinical neurophysiology 9, 4 (1992), 456--479.

[23]

John Polich. 2007. Updating P300: an integrative theory of P3a and P3b. Clinical neurophysiology 118, 10 (2007), 2128--2148.

[24]

J. J. Rocchio. 1971. Relevance Feedback in Information Retrieval. Prentice Hall, Englewood, Cliffs, New Jersey.

[25]

Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. 2017. Scribbler: Controlling Deep Image Synthesis With Sketch and Color. In Proc. CVPR.

[26]

Concetto Spampinato, Simone Palazzo, Isaak Kavasidis, Daniela Giordano, Nasim Souly, and Mubarak Shah. 2017. Deep Learning Human Mind for Automated Visual Classifcation. In Proc. CVPR.

[27]

Michiel Spapé, Rinus Verdonschot, and Henk van Steenbergen. 2019. The E-Primer: An introduction to creating psychological experiments in E-Prime. Second edition updated for E-Prime 3. (2 ed.). Leiden Univerisity Press.

[28]

Nancy K Squires, Kenneth C Squires, and Steven A Hillyard. 1975. Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Deux varieties d'ondes positives de longue latence evoquees par des stimuli auditifs non predictibles chez l'homme. Electroencephalography and Clinical Neurophysiology 38, 4 (1975), 387--401.

[29]

Samuel Sutton, Patricia Tueting, Joseph Zubin, and E. Roy John. 1967. Information delivery and the sensory evoked potential. Science 155, 3768 (1967), 1436--1439.

[30]

Praveen Tirupattur, Yogesh Singh Rawat, Concetto Spampinato, and Mubarak Shah. 2018. ThoughtViz: Visualizing Human Thoughts Using Generative Adversarial Network. In Proc. MM. ACM, New York, NY, USA, 950--958.

Digital Library

[31]

Anne M Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive psychology 12, 1 (1980), 97--136.

[32]

Antti Ukkonen, Pyry Joona, and Tuukka Ruotsalo. 2020. Generating Images Instead of Retrieving Them: Relevance Feedback on Generative Adversarial Networks. In Proc. SIGIR. ACM, New York, NY, USA, 1329--1338.

Digital Library

[33]

Rolf Verleger, Piotr Ja´skowski, and Edmund Wascher. 2005. Evidence for an integrative role of P3b in linking reaction to perception. Journal of Psychophysiology 19, 3 (2005), 165--181.

[34]

Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proc. ICML. JMLR.org, 2048--2057.

[35]

Thorsten O Zander, Laurens R Krol, Niels P Birbaumer, and Klaus Gramann. 2016. Neuroadaptive technology enables implicit cursor control based on medial prefrontal cortex activity. Proceedings of the National Academy of Sciences 113, 52 (2016), 14898--14903.

[36]

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas. 2017. StackGAN: Text to Photo-Realistic Image Synthesis With Stacked Generative Adversarial Networks. In Proc. ICCV.

[37]

Xiang Sean Zhou and Thomas S Huang. 2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia systems 8, 6 (2003), 536--544.

Cited By

de la Torre-Ortiz CRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Perceptual Visual Similarity from EEG: Prediction and Image GenerationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3685508(11146-11155)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3685508
Ma JRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Cognition-Supervised Saliency Detection: Contrasting EEG Signals and Visual StimuliProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681037(7744-7753)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681037
Wei JXia DXie HChang CLi CYang X(2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645211
Show More Cited By

Index Terms

Brain Relevance Feedback for Interactive Image Generation
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Non-invasive brain-actuated interaction
BVAI'07: Proceedings of the 2nd international conference on Advances in brain, vision and artificial intelligence

The promise of Brain-Computer Interfaces (BCI) technology is to augment human capabilities by enabling interaction with computers through a conscious and spontaneous modulation of the brainwaves after a short training period. Indeed, by analyzing brain ...
Advanced brain computer interface for communication and control
AVI '10: Proceedings of the International Conference on Advanced Visual Interfaces

The brain computer interface (BCI) technology allows a direct connection between brain and computer without any muscular activity required, and thus it offers a unique opportunity to enhance and/or to restore communication and actions into external word ...
Optimizing the number of electrodes and spatial filters for Brain-Computer Interfaces by means of an evolutionary multi-objective approach

Optimization of channel selection and spatial filter for Brain Computer Interfaces.A multi-objective evolutionary algorithm simultaneously optimizes filter and channels.Multi-objective approach returns a set of solutions (channels vs. error tradeoffs).A ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

October 2020

1297 pages

ISBN:9781450375146

DOI:10.1145/3379337

General Chairs:
Shamsi Iqbal
Microsoft Research, USA
,
Karon MacLean
University of British Columbia, Canada
,
Program Chairs:
Fanny Chevalier
University of Toronto, Canada
,
Stefanie Mueller
MIT CSAIL, USA

Copyright © 2020 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Academy of Finland

Conference

UIST '20

Sponsor:

UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology

October 20 - 23, 2020

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
454
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)7

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

de la Torre-Ortiz CRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Perceptual Visual Similarity from EEG: Prediction and Image GenerationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3685508(11146-11155)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3685508
Ma JRuotsalo TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Cognition-Supervised Saliency Detection: Contrasting EEG Signals and Visual StimuliProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681037(7744-7753)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681037
Wei JXia DXie HChang CLi CYang X(2024)SpaceEditing: A Latent Space Editing Interface for Integrating Human Knowledge into Deep Neural NetworksProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645211(489-503)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645211
de la Torre-Ortiz CSpapé MRavaja NRuotsalo T(2024)Cross-Subject EEG Feedback for Implicit Image GenerationIEEE Transactions on Cybernetics10.1109/TCYB.2024.340615954:10(6105-6117)Online publication date: Oct-2024
https://doi.org/10.1109/TCYB.2024.3406159
McGuire NMoshfeghi Y(2024)What Song Am I Thinking Of?Machine Learning, Optimization, and Data Science10.1007/978-3-031-53966-4_31(418-432)Online publication date: 15-Feb-2024
https://doi.org/10.1007/978-3-031-53966-4_31
Kato AHorie R(2023)A Study on EEG Signal Features Reflecting Shapes and Colors of Simple Visual Images and their Discrimination視認する単純な画像の形と色を反映する脳波信号特徴量とその判別の研究IEEJ Transactions on Electronics, Information and Systems10.1541/ieejeiss.143.397143:4(397-405)Online publication date: 1-Apr-2023
https://doi.org/10.1541/ieejeiss.143.397
Michalkova DRodriguez MMoshfeghi Y(2023)Understanding Feeling-of-Knowing in Information Search: An EEG StudyACM Transactions on Information Systems10.1145/361138442:3(1-30)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3611384
Ruotsalo TMäkelä KSpapé MLeiva LChen HDuh WHuang HKato MMothe JPoblete B(2023)Affective Relevance: Inferring Emotional Responses via fNIRS NeuroimagingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591946(1796-1800)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591946
Spape MDavis KKangassalo LRavaja NSovijarvi-Spape ZRuotsalo T(2023)Brain-Computer Interface for Generating Personally Attractive ImagesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.305904314:1(637-649)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TAFFC.2021.3059043
Michalkova DParra-Rodriguez MMoshfeghi YAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Information Need AwarenessProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531999(610-621)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531999
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten