research-article

Integrating neurophysiologic relevance feedback in intent modeling for information retrieval

Authors:

Giulio Jacucci,

Tuukka Ruotsalo,

Patrik Pluchino,

Jonathan Freeman,

Luciano Gamberini,

Benjamin BlankertzAuthors Info & Claims

Journal of the Association for Information Science and Technology, Volume 70, Issue 9

Pages 917 - 930

https://doi.org/10.1002/asi.24161

Published: 02 August 2019 Publication History

Abstract

The use of implicit relevance feedback from neurophysiology could deliver effortless information retrieval. However, both computing neurophysiologic responses and retrieving documents are characterized by uncertainty because of noisy signals and incomplete or inconsistent representations of the data. We present the first‐of‐its‐kind, fully integrated information retrieval system that makes use of online implicit relevance feedback generated from brain activity as measured through electroencephalography (EEG), and eye movements. The findings of the evaluation experiment (N = 16) show that we are able to compute online neurophysiology‐based relevance feedback with performance significantly better than chance in complex data domains and realistic search tasks. We contribute by demonstrating how to integrate in interactive intent modeling this inherently noisy implicit relevance feedback combined with scarce explicit feedback. Although experimental measures of task performance did not allow us to demonstrate how the classification outcomes translated into search task performance, the experiment proved that our approach is able to generate relevance feedback from brain signals and eye movements in a realistic scenario, thus providing promising implications for future work in neuroadaptive information retrieval (IR).

References

[1]

Acqualagna, L., Botrel, L., Vidaurre, C., Kübler, A., & Blankertz, B. (2016). Large‐scale assessment of a fully automatic co‐adaptive motor imagery‐based brain computer interface. PLoS One, 11(2), e0148886. https://doi.org/10.1371/journal.pone.0148886

[2]

Agrawal, S., & Goyal, N. (2013). Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proceedings of the 30th International Conference on Machine Learning (pp. 127–135). PMLR.

[3]

Allegretti, M., Moshfeghi, Y., Hadjigeorgieva, M., Pollick, F. E., Jose, J. M., & Pasi, G. (2015). When relevance judgement is happening?: An EEG‐based study. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 719–722).

[4]

Allison, B., Luth, T., Valbuena, D., Teymourian, A., Volosyak, I., & Graser, A. (2010). BCI demographics: How many (and what kinds of) people can use an SSVEP BCI? IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(2), 107–116.

[5]

Arapakis, I., Athanasakos, K., & Jose, J.M. (2010). A comparison of general vs personalised affective models for the prediction of topical relevance. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 371–378). New York, NY: ACM. https://doi.org/10.1145/1835449.1835512

Digital Library

[6]

Arapakis, I., Konstas, I., & Jose, J.M. (2009). Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In Proceedings of the 17th ACM International Conference on Multimedia (pp. 461–470). New York, NY: ACM. https://doi.org/10.1145/1631272.1631336

Digital Library

[7]

Attias, H. (1999). Inferring parameters and structure of latent variable models by variational bayes. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 21–30). San Francisco, CA: Morgan Kaufmann Publishers Inc.

Digital Library

[8]

Baccino, T., & Manunta, Y. (2005). Eye‐fixation‐related potentials: Insight into parafoveal processing. Journal of Psychophysiology, 19(3), 204–215.

[9]

Barral, O., Eugster, M.J., Ruotsalo, T., Spapé, M.M., Kosunen, I., Ravaja, N., … Jacucci, G. (2015). Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In Proceedings of the 20th International Conference on Intelligent User Interfaces (pp. 389–399). New York, NY: ACM. https://doi.org/10.1145/2678025.2701389

Digital Library

[10]

Barral, O., Kosunen, I., & Jacucci, G. (2017). No need to laugh out loud: Predicting humor appraisal of comic strips based on physiological signals in a realistic environment. ACM Transactions on Computer‐Human Interaction, 24(6), 40. Retrieved from http://doi.acm.org/10.1145/3157730. https://doi.org/10.1145/3157730

[11]

Barral, O., Kosunen, I., Ruotsalo, T., Spapé, M.M., Eugster, M.J.A., Ravaja, N., … Jacucci, G. (2016). Extracting relevance and affect information from physiological text annotation. User Modeling and User‐Adapted Interaction, 26(5), 493–520. https://doi.org/10.1007/s11257-016-9184-8

Digital Library

[12]

Blankertz, B., Lemm, S., Treder, M., Haufe, S., & Müller, K.‐R. (2011). Single‐trial analysis and classification of ERP components – A tutorial. NeuroImage, 56(2), 814–825. https://doi.org/10.1016/j.neuroimage.2010.06.048

[13]

Blankertz, B., Sannelli, C., Halder, S., Hammer, E.M., Kübler, A., Müller, K.‐R., … Dickhaus, T. (2010). Neurophysiological predictor of SMR‐based BCI performance. NeuroImage, 51(4), 1303–1309. https://doi.org/10.1016/j.neuroimage.2010.03.022

[14]

Cowley, B., Filetti, M., Lukander, K., Torniainen, J., Henelius, A., Ahonen, L., … Jacucci, G. (2016). The psychophysiology primer: A guide to methods and a broad review with a focus on human–computer interaction. Foundations and Trends[textregistered] in Human—Computer Interaction, 9(3‐4), 151–308. Retrieved from https://doi.org/10.1561/1100000065. https://doi.org/10.1561/1100000065

[15]

Daee, P., Pyykkö, J., Glowacka, D., & Kaski, S. (2016). Interactive intent modeling from multiple feedback domains. In Proceedings of the 21st International Conference on Intelligent User Interfaces (pp. 71–75). New York, NY: ACM. https://doi.org/10.1145/2856767.2856803

Digital Library

[16]

Eugster, M.J.A., Ruotsalo, T., Spapé, M.M., Barral, O., Ravaja, N., Jacucci, G., & Kaski, S. (2016). Natural brain‐information interfaces: Recommending information by relevance inferred from human brain signals. Scientific Reports, 6, 38580. https://doi.org/10.1038/srep38580

[17]

Eugster, M.J.A., Ruotsalo, T., Spapé, M.M., Kosunen, I., Barral, O., Ravaja, N., … Kaski, S. (2014). Predicting term‐relevance from brain signals. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (pp. 425–434). New York, NY: ACM. https://doi.org/10.1145/2600428.2609594

Digital Library

[18]

Farwell, L.A., & Donchin, E. (1988). Talking off the top of your head: Toward a mental prosthesis utilizing event‐related brain potentials. Electroencephalography and clinical Neurophysiology, 70(6), 510–523.

[19]

Friedman, J.H. (1989). Regularized discriminant analysis. Journal of the American Statistical Association, 84(405), 165–175. https://doi.org/10.1080/01621459.1989.10478752

[20]

Golenia, J.‐E., Wenzel, M.A., & Blankertz, B. (2015). Live demonstrator of EEG and eye‐tracking input for disambiguation of image search results. In Symbiotic interaction (pp. 81–86). Cham: Springer. https://doi.org/10.1007/978-3-319-24917-9_8

[21]

Golenia, J.E., Wenzel, M.A., Bogojeski, M., & Blankertz, B. (2018). Implicit relevance feedback from electroencephalography and eye tracking in image search. Journal of Neural Engineering, 15(2), 026002. https://doi.org/10.1088/1741-2552/aa9999

[22]

Good, P. (2013). Permutation tests: a practical guide to resampling methods for testing hypotheses. Berlin/Heidelberg, Germany: Springer Science & Business Media.

[23]

Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., … Edlinger, G. (2009). How many people are able to control a p300‐based brain–computer interface (BCI)? Neuroscience Letters, 462(1), 94–98.

[24]

Gwizdka, J. (2014). Characterizing relevance with eye‐tracking measures. In Proceedings of the 5th Information Interaction in Context Symposium (pp. 58–67). New York, NY: ACM. https://doi.org/10.1145/2637002.2637011

Digital Library

[25]

Gwizdka, J., & Mostafa, J. (2015). NeuroIR 2015: SIGIR 2015 workshop on neuro‐physiological methods in IR research. In ACM SIGIR Forum (Vol. 49, pp. 83–88). ACM. https://doi.org/10.1145/2888422.2888435

[26]

Gwizdka, J., & Mostafa, J. (2017). NeuroIIR: Challenges in bringing neuroscience to research in human‐information interaction. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval ‐ CHIIR ’17 (pp. 437–438). New York, NY: ACM Press. https://doi.org/10.1145/3020165.3022165

[27]

Jacucci, G., Fairclough, S., & Solovey, E.T. (2015). Physiological computing. Computer, 48(10), 12–16. Retrieved from http://ieeexplore.ieee.org/document/7310960/. https://doi.org/10.1109/MC.2015.291

Digital Library

[28]

Kangasrääsiö, A., Chen, Y., Glowacka, D., & Kaski, S. (2016). Interactive modeling of concept drift and errors in relevance feedback. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization (pp. 185–193). New York, NY: ACM. https://doi.org/10.1145/2930238.2930243

Digital Library

[29]

Kauppi, J.‐P., Kandemir, M., Saarinen, V.‐M., Hirvenkari, L., Parkkonen, L., Klami, A., … Kaski, S. (2015). Towards brain‐activity‐controlled information retrieval: Decoding image relevance from MEG signals. NeuroImage, 112, 288–298. https://doi.org/10.1016/j.neuroimage.2014.12.079

[30]

Kelly, D., & Fu, X. (2006). Elicitation of term relevance feedback: An investigation of term source and context. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 453–460). New York, NY: ACM. https://doi.org/10.1145/1148170.1148249

[31]

Kelly, D., & Teevan, J. (2003). Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2), 18–28. https://doi.org/10.1145/959258.959260

Digital Library

[32]

Ledoit, O., & Wolf, M. (2004). A well‐conditioned estimator for large‐dimensional covariance matrices. Journal of Multivariate Analysis, 88(2), 365–411. https://doi.org/10.1016/S0047-259X(03)00096-4

Digital Library

[33]

Lemm, S., Blankertz, B., Dickhaus, T., & Müller, K.‐R. (2011). Introduction to machine learning for brain imaging. Neuroimage, 56(2), 387–399.

[34]

Moshfeghi, Y., & Jose, J.M. (2013). An effective implicit relevance feedback technique using affective, physiological and behavioural features. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 133–142). New York, NY: ACM. https://doi.org/10.1145/2484028.2484074

Digital Library

[35]

Moshfeghi, Y., Pinto, L.R., Pollick, F.E., & Jose, J.M. (2013). Understanding relevance: An fMRI study. In European Conference on Information Retrieval (pp. 14–25). Berlin, Heidelberg: Springer.

[36]

Mostafa, J., & Gwizdka, J. (2016). Deepening the role of the user: Neuro‐physiological evidence as a basis for studying and improving search. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (pp. 63–70). New York, USA; ACM.

[37]

Nijholt, A., Tan, D., Pfurtscheller, G., Brunner, C., Millán, J.d.R., Allison, B., … Müller, K.R. (2008). Brain‐computer interfacing for intelligent systems. IEEE Intelligent Systems, 23(3), 72–79. https://doi.org/10.1109/MIS.2008.41

Digital Library

[38]

Oliveira, F.T., Aula, A., & Russell, D.M. (2009). Discriminating the relevance of web search results with measures of pupil size. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2209–2212). New York, NY: ACM. https://doi.org/10.1145/1518701.1519038

Digital Library

[39]

Puolamäki, K., Salojärvi, J., Savia, E., Simola, J., & Kaski, S. (2005). Combining eye movements and collaborative filtering for proactive information retrieval. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 146–153). New York, USA; ACM.

Digital Library

[40]

Ruotsalo, T., Jacucci, G., Myllymäki, P., & Kaski, S. (2015). Interactive intent modeling: Information discovery beyond search. Communications of the ACM, 58(1), 86–92.

Digital Library

[41]

Ruotsalo, T., Peltonen, J., Eugster, M., Głowacka, D., Konyushkova, K., Athukorala, K., et al. (2013). Directing exploratory search with interactive intent modeling. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (pp. 1759–1764). New York, USA; ACM.

Digital Library

[42]

Schäfer, J., & Strimmer, K. (2005). A shrinkage approach to large‐scale covariance matrix estimation and implications for functional genomics. Statistical Applications in Genetics and Molecular Biology, 4(1). Article 32. https://doi.org/10.2202/1544-6115.1175

[43]

Treder, M.S., Schmidt, N.M., & Blankertz, B. (2011). Gaze‐independent brain–computer interfaces based on covert attention and feature attention. Journal of Neural Engineering, 8(6), 066003.

[44]

Wenzel, M.A., Bogojeski, M., & Blankertz, B. (2017). Real‐time inference of word relevance from electroencephalogram and eye gaze. Journal of Neural Engineering, 14(5), 056007. https://doi.org/10.1088/1741-2552/aa7590

[45]

Wenzel, M.A., Golenia, J.‐E., & Blankertz, B. (2016). Classification of eye fixation related potentials for variable stimulus saliency. Frontiers in Neuroscience, 10, 23. Retrieved from https://www.frontiersin.org/article/10.3389/fnins.2016.00023. https://doi.org/10.3389/fnins.2016.00023

[46]

Zhu, X., & Davidson, I. (2007). Knowledge discovery and data mining: Challenges and realities. Hershey, PA: IGI Global.

Cited By

Michalkova DRodriguez MMoshfeghi Y(2023)Understanding Feeling-of-Knowing in Information Search: An EEG StudyACM Transactions on Information Systems10.1145/361138442:3(1-30)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3611384
Ruotsalo TMäkelä KSpapé MLeiva LChen HDuh WHuang HKato MMothe JPoblete B(2023)Affective Relevance: Inferring Emotional Responses via fNIRS NeuroimagingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591946(1796-1800)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591946
Spapé MDavis KKangassalo LRavaja NSovijärvi-Spapé ZRuotsalo T(2023)Brain-Computer Interface for Generating Personally Attractive ImagesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.305904314:1(637-649)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3059043
Show More Cited By

Index Terms

Integrating neurophysiologic relevance feedback in intent modeling for information retrieval
1. Information systems
  1. Information retrieval

Index terms have been assigned to the content through auto-classification.

Recommendations

Contextual relevance feedback in web information retrieval
IIiX: Proceedings of the 1st international conference on Information interaction in context

In this paper, we present an alternative approach to the problem of contextual relevance feedback in web-based information retrieval. Our approach utilises a rich contextual model that exploits a user's implicit and explicit data. Each user's implicit ...
Adaptive relevance feedback in information retrieval
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Relevance Feedback has proven very effective for improving retrieval accuracy. A difficult yet important problem in all relevance feedback methods is how to optimally balance the original query and feedback information. In the current feedback methods, ...
Information retrieval with concept-based pseudo-relevance feedback in MEDLINE

Although using domain specific knowledge sources for information retrieval yields more accurate results compared to pure keyword-based methods, more improvements can be achieved by considering both relations between concepts in an ontology and also their ...

Comments

Information & Contributors

Information

Published In

cover image Journal of the Association for Information Science and Technology

Journal of the Association for Information Science and Technology Volume 70, Issue 9

September 2019

136 pages

ISSN:2330-1635

EISSN:2330-1643

DOI:10.1002/asi.v70.9

Issue’s Table of Contents

© 2019 The Authors. Journal of the Association for Information Science and Technology published by Wiley Periodicals, Inc. on behalf of ASIS&T.

This is an open access article under the terms of the Creative Commons Attribution‐NonCommercial‐NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non‐commercial and no modifications or adaptations are made.

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 02 August 2019

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Michalkova DRodriguez MMoshfeghi Y(2023)Understanding Feeling-of-Knowing in Information Search: An EEG StudyACM Transactions on Information Systems10.1145/361138442:3(1-30)Online publication date: 30-Oct-2023
https://dl.acm.org/doi/10.1145/3611384
Ruotsalo TMäkelä KSpapé MLeiva LChen HDuh WHuang HKato MMothe JPoblete B(2023)Affective Relevance: Inferring Emotional Responses via fNIRS NeuroimagingProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591946(1796-1800)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591946
Spapé MDavis KKangassalo LRavaja NSovijärvi-Spapé ZRuotsalo T(2023)Brain-Computer Interface for Generating Personally Attractive ImagesIEEE Transactions on Affective Computing10.1109/TAFFC.2021.305904314:1(637-649)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3059043
McGuire NMoshfeghi Y(2023)What Song Am I Thinking Of?Machine Learning, Optimization, and Data Science10.1007/978-3-031-53966-4_31(418-432)Online publication date: 22-Sep-2023
https://dl.acm.org/doi/10.1007/978-3-031-53966-4_31
Ye ZXie XLiu YWang ZChen XZhang MMa S(2022)Towards a Better Understanding of Human Reading Comprehension with Brain SignalsProceedings of the ACM Web Conference 202210.1145/3485447.3511966(380-391)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3485447.3511966
Michalkova DParra-Rodriguez MMoshfeghi YAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Information Need AwarenessProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531999(610-621)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531999
Jacucci GDaee PVuong TAndolina SKlouche KSjöberg MRuotsalo TKaski S(2021)Entity Recommendation for Everyday Digital TasksACM Transactions on Computer-Human Interaction10.1145/345891928:5(1-41)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3458919
de la Torre-Ortiz CSpapé MKangassalo LRuotsalo TIqbal SMacLean KChevalier FMueller S(2020)Brain Relevance Feedback for Interactive Image GenerationProceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379337.3415821(1060-1070)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3379337.3415821
Davis KKangassalo LSpapé MRuotsalo TBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Brainsourcing: Crowdsourcing Recognition Tasks via Collaborative Brain-Computer InterfacingProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376288(1-14)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376288
Gwizdka JMoshfeghi YWilson M(2019)Introduction to the special issue on neuro‐information scienceJournal of the Association for Information Science and Technology10.1002/asi.2426370:9(911-916)Online publication date: 2-Aug-2019
https://dl.acm.org/doi/10.1002/asi.24263

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents