column

Research Frontiers in Information Retrieval: Report from the Third Strategic Workshop on Information Retrieval in Lorne (SWIRL 2018)

Authors:

J. Shane Culpepper,

Mark D. SmuckerAuthors Info & Claims

ACM SIGIR Forum, Volume 52, Issue 1

Pages 34 - 90

https://doi.org/10.1145/3274784.3274788

Published: 31 August 2018 Publication History

Abstract

The purpose of the Strategic Workshop in Information Retrieval in Lorne is to explore the long-range issues of the Information Retrieval field, to recognize challenges that are on - or even over - the horizon, to build consensus on some of the key challenges, and to disseminate the resulting information to the research community. The intent is that this description of open problems will help to inspire researchers and graduate students to address the questions, and will provide funding agencies data to focus and coordinate support for information retrieval research.

References

[1]

Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 65-74, New York, NY, USA, 2017. ACM.

Digital Library

[2]

Karl Moritz Hermann, Tomá? Ko?isk?, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA, 2015. MIT Press.

Digital Library

[3]

Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple NLP tasks. CoRR, abs/1611.01587, 2016.

[4]

Bhaskar Mitra and Nick Craswell. An introduction to neural information retrieval. Foundations and Trends in Information Retrieval. Now Publishers Inc., 2018.

[5]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013.

[6]

Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. CoRR, abs/1707.05589, 2017.

[7]

Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 1291-1299, Republic and Canton of Geneva, Switzerland, 2017. International World Wide Web Conferences Steering Committee.

Digital Library

[8]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119. Curran Associates, Inc., 2013.

Digital Library

[9]

Ryan Spring and Anshumali Shrivastava. Scalable and sustainable deep learning via randomized hashing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pages 445-454, New York, NY, USA, 2017. ACM.

Digital Library

[10]

Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS'15, pages 2440-2448, Cambridge, MA, USA, 2015. MIT Press.

Digital Library

[11]

Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. CoRR, abs/1708.08296, 2017.

[12]

Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. CoRR, abs/1410.3916, 2014.

[13]

Chenyan Xiong, Jamie Callan, and Tie-Yan Liu. Word-entity duet representations for document ranking. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 763-772, New York, NY, USA, 2017. ACM.

Digital Library

[14]

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 55-64, New York, NY, USA, 2017. ACM.

Digital Library

[15]

Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pages 2397-2406. JMLR.org, 2016.

Digital Library

[16]

Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. Towards conversational recommender systems. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 815-824, New York, NY, USA, 2016. ACM.

Digital Library

[17]

Ido Guy. Searching by talking: Analysis of voice queries on mobile web search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 35-44, New York, NY, USA, 2016. ACM.

Digital Library

[18]

Brent Hecht, Jaime Teevan, Meredith Ringe Morris, and Dan Liebling. Searchbuddies: Bringing search engines into the conversation. In ICWSM 2012 - Proceedings of the 6th International AAAI Conference on Weblogs and Social Media, pages 138-145, 12 2012.

[19]

Jiepu Jiang, Ahmed Hassan Awadallah, Rosie Jones, Umut Ozertem, Imed Zitouni, Ranjitha Gurunath Kulkarni, and Omar Zia Khan. Automatic online evaluation of intelligent assistants. In Proceedings of the 24th International Conference on World Wide Web, WWW '15, pages 506-516, Republic and Canton of Geneva, Switzerland, 2015. International World Wide Web Conferences Steering Committee.

Digital Library

[20]

Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202. Association for Computational Linguistics, 2016.

[21]

Filip Radlinski and Nick Craswell. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR '17, pages 117-126, New York, NY, USA, 2017. ACM.

Digital Library

[22]

Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pages 3776-3783. AAAI Press, 2016.

Digital Library

[23]

Alexandra Vtyurina, Denis Savenkov, Eugene Agichtein, and Charles L. A. Clarke. Exploring conversational search with humans, assistants, and wizards. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA '17, pages 2187-2193, New York, NY, USA, 2017. ACM.

Digital Library

[24]

Tsung-Hsien Wen, David Vandyke, Nikola Mrk?i?, Milica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449. Association for Computational Linguistics, 2017.

[25]

Rui Yan, Yiping Song, and Hua Wu. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 55-64, New York, NY, USA, 2016. ACM.

Digital Library

[26]

Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4349-4357. Curran Associates, Inc., 2016.

Digital Library

[27]

Solon Barocas and Andrew D. Selbst. Big data's disparate impact. California Law Review, 104, 2016.

[28]

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186, April 2017.

[29]

Amit Datta, Michael Carl Tschantz, and Anupam Datta. Automated experiments on ad privacy settings. PoPETs, 2015(1):92-112, 2015.

[30]

Carsten Eickhoff. Cognitive biases in crowdsourcing. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, pages 162-170, New York, NY, USA, 2018. ACM.

Digital Library

[31]

Lilli Japec, Frauke Kreuter, Marcus Berg, Paul Biemer, Paul Decker, Cliff Lampe, Julia Lane, Cathy O'Neil, and Abe Usher. Big data in survey researchaapor task force report. Public Opinion Quarterly, 79(4):839-880, 2015.

[32]

Rishabh Mehrotra, Amit Sharma, Ashton Anderson, Fernando Diaz, Hanna Wallach, and Emine Yilmaz. Auditing search engines for differential satisfaction across demographics. In Proceedings of the 26th International Conference on World Wide Web, 2017.

Digital Library

[33]

Jahna Otterbacher, Jo Bates, and Paul Clough. Competent men and warm women: Gender stereotypes and backlash in image search results. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pages 6620-6631, New York, NY, USA, 2017. ACM.

Digital Library

[34]

Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. Social data: Biases, methodological pitfalls, and ethical boundaries. Available at SSRN, 2017.

[35]

Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pages 687-696, New York, NY, USA, 2017. ACM.

Digital Library

[36]

Léon Bottou, Jonas Peters, Joaquin Qui nonero Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14:3207-3260, 2013.

Digital Library

[37]

Ben Carterette. Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM Trans. Inf. Syst., 30(1):4:1-4:34, March 2012.

Digital Library

[38]

Ben Carterette. Bayesian inference for information retrieval evaluation. In Proceedings of the 2015 International Conference on The Theory of Information Retrieval, ICTIR '15, pages 31-40, New York, NY, USA, 2015. ACM.

Digital Library

[39]

Ben Carterette. The best published result is random: Sequential testing and its effect on reported effectiveness. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 747-750, New York, NY, USA, 2015. ACM.

Digital Library

[40]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. Unbiased learning-to-rank with biased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM '17, pages 781-789, New York, NY, USA, 2017. ACM.

Digital Library

[41]

Alistair Moffat, Peter Bailey, Falk Scholer, and Paul Thomas. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM Transactions on Information Systems (TOIS), 35(3):24, 2017.

Digital Library

[42]

Alistair Moffat and Justin Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS), 27(1):2, 2008.

Digital Library

[43]

Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251), 2015.

[44]

Tetsuya Sakai. Statistical significance, power, and sample sizes: A systematic review of sigir and tois, 2006-2015. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 5-14, New York, NY, USA, 2016. ACM.

Digital Library

[45]

Tetsuya Sakai. Topic set size design. Inf. Retr., 19(3):256-283, June 2016.

Digital Library

[46]

Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3632-3642. Curran Associates, Inc., 2017.

[47]

Ellen M. Voorhees, Daniel Samarov, and Ian Soboroff. Using replicates in information retrieval evaluation. ACM Trans. Inf. Syst., 36(2):12:1-12:21, August 2017.

Digital Library

[48]

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 115-124, New York, NY, USA, 2016. ACM.

Digital Library

[49]

Xiao Bai, B. Barla Cambazoglu, and Archie Russell. Improved caching techniques for large-scale image hosting services. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '16, pages 639-648, New York, NY, USA, 2016. ACM.

Digital Library

[50]

Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg. Off the beaten path: Let's replace term-based retrieval with k-nn search. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 1099-1108. ACM, 2016.

Digital Library

[51]

Ruey-Cheng Chen, Luke Gallagher, Roi Blanco, and J Shane Culpepper. Efficient cost-aware cascade ranking in multi-stage retrieval. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 445-454. ACM, 2017.

Digital Library

[52]

Bob Goodwin, Michael Hopcroft, Dan Luu, Alex Clemmer, Mihaela Curmei, Sameh Elnikety, and Yuxiong He. Bitfunnel: Revisiting signatures for search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 605-614, New York, NY, USA, 2017. ACM.

Digital Library

[53]

Tim Kraska, Alex Beutel, Ed H. Chi, Jeff Dean, and Neoklis Polyzotis. The case for learned index structures. 2017. ArXiV - https://arxiv.org/abs/1712.01208.

[54]

Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. Quickscorer: A fast algorithm to rank documents with additive ensembles of regression trees. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 73-82, New York, NY, USA, 2015. ACM.

Digital Library

[55]

Giuseppe Ottaviano and Rossano Venturini. Partitioned elias-fano indexes. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '14, pages 273-282, New York, NY, USA, 2014. ACM.

Digital Library

Cited By

Bernard NBalog KOosterhuis HBast HXiong C(2024)Towards a Formal Characterization of User Simulation Objectives in Conversational Information AccessProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672529(185-193)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672529
Fu XPerez-Ortiz MLipani AOosterhuis HBast HXiong C(2024)An Analysis of Stopping Strategies in Conversational Search SystemsProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672524(247-257)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672524
Wu XWan HTan QYao WLiu N(2024)DIRECT: Dual Interpretable Recommendation with Multi-aspect Word AttributionACM Transactions on Intelligent Systems and Technology10.1145/3663483Online publication date: 6-May-2024
https://doi.org/10.1145/3663483
Show More Cited By

Index Terms

Research Frontiers in Information Retrieval: Report from the Third Strategic Workshop on Information Retrieval in Lorne (SWIRL 2018)
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne

During a three-day workshop in February 2012, 45 Information Retrieval researchers met to discuss long-range challenges and opportunities within the field. The result of the workshop is a diverse set of research directions, project ideas, and challenge ...
Research on Content-Based Multimedia Information Retrieval
ICCIS '11: Proceedings of the 2011 International Conference on Computational and Information Sciences

This paper firstly studies the features of content-based multimedia information retrieval, then analyzes the system structure of content-based multimedia information retrieval and retrieval procedures, lastly the author discusses the exist limitations ...
Advanced Information Retrieval

In this paper we explore some of the most important areas of advanced information retrieval. In particular, we look at cross-lingual information retrieval, multimedia information retrieval and semantic-based information retrieval. Cross-lingual ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGIR Forum

ACM SIGIR Forum Volume 52, Issue 1

June 2018

167 pages

ISSN:0163-5840

DOI:10.1145/3274784

Editors:
Claudia Hauff
Delft University of Technology. The Netherlands
,
Craig Macdonald
University of Glasgow, Glasgow, UK

Issue’s Table of Contents

Copyright © 2018 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 August 2018

Published in SIGIR Volume 52, Issue 1

Check for updates

Qualifiers

Column

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

131
Total Citations
View Citations
765
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)11

Reflects downloads up to 13 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bernard NBalog KOosterhuis HBast HXiong C(2024)Towards a Formal Characterization of User Simulation Objectives in Conversational Information AccessProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672529(185-193)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672529
Fu XPerez-Ortiz MLipani AOosterhuis HBast HXiong C(2024)An Analysis of Stopping Strategies in Conversational Search SystemsProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672524(247-257)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672524
Wu XWan HTan QYao WLiu N(2024)DIRECT: Dual Interpretable Recommendation with Multi-aspect Word AttributionACM Transactions on Intelligent Systems and Technology10.1145/3663483Online publication date: 6-May-2024
https://doi.org/10.1145/3663483
Frummet APapenmeier AFröbe MKiesel J(2024)The Eighth Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI’24)Proceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638310(433-435)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638310
Li YCapra R(2024)Stopped yet Completed: Exploring the Relationships between Session-stopping Reasons, Information Types, and Cognitive Activities in Cross-Session SearchesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638304(119-129)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638304
Gienapp LScells HDeckers NBevendorff JWang SKiesel JSyed SFröbe MZuccon GStein BHagen MPotthast MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Generative Ad Hoc Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657849(1916-1929)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657849
Bernard NAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Leveraging User Simulation to Develop and Evaluate Conversational Information Access AgentsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635730(1136-1138)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635730
Łajewska WAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Grounded and Transparent Response Generation for Conversational Information-Seeking SystemsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635727(1142-1144)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635727
Wang ZXu ZSrikumar VAi QChua TNgo CKa-Wei Lee RKumar RLauw H(2024)An In-depth Investigation of User Response Simulation for Conversational SearchProceedings of the ACM Web Conference 202410.1145/3589334.3645447(1407-1418)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645447
Bellamy EFarrell KHopping APinter JSaju MBeskow D(2024)Designing an Intelligent System to Map Global Connections2024 IEEE International Systems Conference (SysCon)10.1109/SysCon61195.2024.10553593(1-3)Online publication date: 15-Apr-2024
https://doi.org/10.1109/SysCon61195.2024.10553593
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents