Evaluation-as-a-Service for the Computational Sciences: Overview and Outlook

Daron Acemoglu, Gino Gancia, and Fabrizio Zilibotti. 2012. Competing engines of growth: Innovation and standardization. J. Econ. Theory 147, 2 (2012), 570--601.e3.

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements that don’t add up: Ad hoc retrieval results since 1998. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). ACM, 601--610.

Michael Arrington. 2006. AOL Proudly Releases Massive Amounts of Private Data. Retrieved from https://techcrunch.com/2006/08/06/aol-proudly-releases-massive-amounts-of-user-search-data/.

Krisztian Balog, Liadh Kelly, and Anne Schuth. 2014. Head first: Living labs for ad hoc search evaluation. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM’14). 1815--1818.

Torben Brodt and Frank Hopfgartner. 2014. Shedding light on a living lab: The CLEF NEWSREEL open recommendation platform. In Proceedings of Information Interaction in Context Conference (IIiX’14). ACM, 223--226.

Jamie Callan and Alistair Moffat. 2012. Panel on use of proprietary data. SIGIR Forum 46, 2 (2012).

Gauthier Chassang. 2017. The impact of the EU general data protection regulation on scientific research. Ecancermedicalscience 11, 709 (Jan. 2017).

Gordon V. Cormack and Thomas R. Lynam. 2005. Spam corpus creation for TREC. In Proceedings of the 2nd Conference on Email and Anti-Spam (CEAS’05).

Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems. Lecture Notes in Computer Science, Vol. 1857. Springer, 1--15.

Nicola Ferro, Norbert Fuhr, Kalervo Järvelin, Noriko Kando, Matthias Lippold, and Justin Zobel. 2016. Increasing reproducibility in IR: Findings from the Dagstuhl seminar on “reproducibility of data-oriented experiments in e-science.”SIGIR Forum 50, 1 (June 2016), 68--82.

Nicola Ferro and Gianmaria Silvello. 2014. CLEF 15th birthday: What can we learn from ad hoc retrieval? In Proceedings of the Conference on Information Access Evaluation. Multilinguality, Multimodality, and Interaction, Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms (Eds.). Lecture Notes in Computer Science, Vol. 8685. Springer, 31--43.

Antonio Foncubierta-Rodríguez and Henning Müller. 2012. Ground truth generation in medical imaging: A crowdsourcing based iterative approach. In Proceedings of the Workshop on Crowdsourcing for Multimedia, ACM Multimedia.

Juliana Freire and Claudio T. Silva. 2012. Making computations and publications reproducible with VisTrails. Comput. Sci. Eng. 14, 4 (Aug. 2012), 18 --25.

Norbert Fuhr. 2017. Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum 51, 3 (2017), 32--41.

Tim Gollub, Benno Stein, and Steven Burrows. 2012. Ousting ivory tower research: Toward a web framework for providing experiments as a service. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’12). ACM, 1125--1126.

Maura R. Grossman and Gordon V. Cormack. 2014. Comments on “the implications of rule 26 (g) on the use of technology-assisted review.”Fed. Cts. L. Rev. 2014 (2014), 285--285.

Allan Hanbury and Henning Müller. 2010. Automated component--level evaluation: Present and future. In Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF’10), Lecture Notes in Computer Science (LNCS), Vol. 6360. Springer, 124--135.

Allan Hanbury, Henning Müller, Krisztian Balog, Torben Brodt, Gordon V. Cormack, Ivan Eggel, Tim Gollub, Frank Hopfgartner, Jayashree Kalpathy-Cramer, Noriko Kando, Anastasia Krithara, Jimmy J. Lin, Simon Mercer, and Martin Potthast. 2015. Evaluation-as-a-service: Overview and outlook. CoRR abs/1512.07454. Retrieved from http://arxiv.org/abs/1512.07454.

Allan Hanbury, Henning Müller, Georg Langs, Marc André Weber, Bjoern H. Menze, and Tomas Salas Fernandez. 2012. Bringing the algorithms to the data: Cloud-based benchmarking for medical image analysis. In Proceedings of the 3rd International Conference of the CLEF Initiative (CLEF’12). Springer Verlag, 24--29.

Donna K. Harman. 1992. Evaluation issues in information retrieval. Info. Process. Manage. 28 (1992), 439--440.

David Hawking. 2015. If SIGIR had an academic track, what would be in it? In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’15). ACM, New York, NY, 1077--1077.

Katja Hofmann, Lihong Li, and Filip Radlinski. 2016. Online evaluation for information retrieval. Found. Trends Info. Retr. 10, 1 (June 2016), 1--117.

Frank Hopfgartner, Allan Hanbury, Henning Mueller, Noriko Kando, Simon Mercer, Jayashree Kalpathy-Cramer, Martin Potthast, Tim Gollub, Anastasia Krithara, Jimmy Lin, Krisztian Balog, and Ivan Eggel. 2015. Report on the evaluation-as-a-service (EaaS) expert workshop. SIGIR Forum 49, 1 (2015), 57--65.

Frank Hopfgartner, Benjamin Kille, Andreas Lommatzsch, Torben Brodt, and Tobias Heintz. 2014. Benchmarking news recommendations in a living lab. In Proceedings of the 5th International Conference of the CLEF Initiative (CLEF’14). Springer Verlag, 250--267.

Frank Hopfgartner, Andreas Lommatzsch, Benjamin Kille, Martha Larson, Torben Brodt, Paolo Cremonesi, and Alexandros Karatzoglou. 2016. The potentials of recommender systems challenges for student learning. In Proceedings of the Challenges in Machine Learning: Gaming and Education (CiML’16).

Xiao Hu, Jin Ha Lee, David Bainbridge, Kahyun Choi, Peter Organisciak, and J. Stephen Downie. 2015. The MIREX grand challenge: A framework of holistic user-experience evaluation in music information retrieval. J. Assoc. Info. Sci. Technol. 68, 1 (2015).

Bernardo Huberman. 2012. Big data deserve a bigger audience. Nature 482 (2012).

Darrel C. Ince, Leslie Hatton, and John Graham-Cumming. 2012. The case for open computer programs. Nature 482, 7386 (Feb. 2012), 485--488.

G. Douglas Jenkins Jr., Atul Mitra, Nina Gupta, and Jason D. Shaw. 1998. Are financial incentives related to performance? A meta-analytic review of empirical research. J. Appl. Psychol. 83, 5 (1998), 777--787.

Makoto P. Kato, Takehiro Yamamoto, Tomohiro Manabe, Akiomi Nishida, and Sumio Fujita. 2017. Overview of the NTCIR-13 OpenLiveQ task. In Proceedings of the 13th NII Testbeds and Community for Information Research Conference (NTCIR’17).

Benjamin Kille, Frank Hopfgartner, Torben Brodt, and Tobias Heintz. 2013. The plista dataset. In Proceedings of the International Workshop and Challenge on News Recommender Systems (NRS’13). ACM, 14--21.

Benjamin Kille, Andreas Lommatzsch, Frank Hopfgartner, Martha Larson, and Arjen P. de Vries. 2017. A stream-based resource for multi-dimensional evaluation of recommender algorithms. In Proceedings of the Special Interest Group on Information Retrieval (SIGIR’17). 1257--1260.

Amy Jo Kim. 2000. Community Building on the Web: Secret Strategies for Successful Online Communities (1st ed.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA.

Rob Kitchin. 2014. The Data Revolution: Big Data, Open Data, Data Infrastructures and Their Consequences. Sage.

Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. 2012. Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 786--794.

Markus Krenn, Matthias Dorfer, Oscar Alfonso Jimenez del Toro, Henning Müller, Bjoern Menze, Marc-Andre Weber, Allan Hanbury, and Georg Langs. 2015. Creating a large-scale silver corpus from multiple algorithmic segmentations. In Proceedings of the Medical Computer Vision Workshop 2015 at MICCAI. LNCS, Vol. 9059. Springer, Munich.

Udo Kruschwitz and Charlie Hall. 2017. Searching the enterprise. Found. Trends Info. Retriev. 11, 1 (2017).

Takuya Kudo. 2010. Creating an age where anyone can find the information they truly need: NTCIR’s information retrieval ideal. NII Today 34 (2010), 4--7.

Mounia Lalmas and Liangjie Hong. 2018. Tutorial on metrics of user engagement: Applications to news, search and e-commerce. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM’18). 781--782.

Georg Langs, Henning Müller, Bjoern H. Menze, and Allan Hanbury. 2012. VISCERAL: Toward large data in medical imaging—Challenges and directions. In Proceedings of the 3rd MICCAI International Workshop (MCBR--CDS’12). Springer, 92--98.

Carol Lefebvre, Eric Manheimer, and Julie Glanville. 2008. Searching for studies. Cochrane Handbook for Systematic Reviews of Interventions, 95--150.

Mark Levy. 2013. Offline evaluation of recommender systems: All pain and no gain? In Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation (RepSys’13). ACM.

David E. Polley Light, Robert P. and Katy Börner. 2013. Open data and open code for big science of science studies. In Proceedings of International Society of Scientometrics and Informetrics Conference. 1342--1356.

Jimmy Lin and Miles Efron. 2013. Overview of the TREC-2013 microblog track. In Proceedings of the 22nd Text REtrieval Conference (TREC’13). Gaithersburg, Maryland.

Andreas Lommatzsch. 2014. Real-time news recommendation using context-aware ensembles. In Advances in Information Retrieval—Proceedings of the 36th European Conference on IR Research (ECIR’14). 51--62.

Andreas Lommatzsch, Benjamin Kille, Frank Hopfgartner, Martha Larson, Torben Brodt, Jonas Seiler, and Özlem Özgöbek. 2017. CLEF 2017 NewsREEL overview: A stream-based recommender task for evaluation and education. In Experimental IR Meets Multilinguality, Multimodality, and Interaction. Springer International Publishing, Cham, 239--254.

James Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, and Angela Hung Byres. 2011. Big Data: The Next Frontier for Innovation, Competition, and Productivity. Technical Report.

John Markoff. 2012. Troves of personal data, forbidden to researchers. The New York Times (21 May 2012).

Nathan Marz and James Warren. 2015. Big Data: Principles and Best Practices of Scalable Real-time Data Systems (1st ed.). Manning Publications Co., Greenwich, CT.

Karl Matthias. 2015. Docker: Up and Running. O’Reilly.

Richard McCreadie, Ian Soboroff, Jimmy Lin, Craig Macdonald, Iadh Ounis, and Dean McCullough. 2012. On building a reusable Twitter corpus. In Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12). 1113--1114.

Henning Müller, Jayashree Kalpathy-Cramer, Allan Hanbury, Keyvan Farahani, Rinat Sergeev, Jin H. Paik, Arno Klein, Antonio Criminisi, Andrew Trister, Thea Norman, David Kennedy, Ganapati Srinivasa, Artem Mamonov, and Nina Preuss. 2016. Report on the cloud--based evaluation approaches workshop 2015. ACM SIGIR Forum 51, 1 (2016), 35--41.

Virginia Ortiz-Repiso, Jane Greenberg, and Javier Calzada-Prado. 2018. A cross-institutional analysis of data-related curricula in information science programmes: A focused look at the iSchools. J. Info. Sci. (2018).

Iadh Ounis, Craig Macdonald, Jimmy Lin, and Ian Soboroff. 2011. Overview of the TREC-2011 microblog track. In Proceedings of the 20th Text REtrieval Conference (TREC’11).

Martin Potthast, Sarah Braun, Tolga Buz, Fabian Duffhauss, Florian Friedrich, Jörg Marvin Gülzow, Jakob Köhler, Winfried Lötzsch, Fabian Müller, Maike Elisa Müller, Robert Paßmann, Bernhard Reinke, Lucas Rettenmeier, Thomas Rometsch, Timo Sommer, Michael Träger, Sebastian Wilhelm, Benno Stein, Efstathios Stamatatos, and Matthias Hagen. 2016. Who wrote the web? Revisiting influential author identification research applicable to information retrieval. In Proceedings of the European Conference on Information Retrieval (ECIR’16), Vol. 9626. Springer, Berlin, 393--407.

Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the reproducibility of PAN’s shared tasks: Plagiarism detection, author identification, and author profiling. In Proceedings of the 5th International Conference of the CLEF Initiative (CLEF’14). Springer Verlag, 268--299.

Martin Potthast, Matthias Hagen, and Benno Stein. 2016. Author obfuscation: Attacking the state of the art in authorship verification. In Proceedings of the CLEF 2016 Evaluation Labs (CEUR’16), Vol. 1609. CLEF and CEUR-WS.org.

Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc (Eds.). 2006. Machine-Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment. Number 3944 in LNAI. Springer.

Jinfeng Rao, Jimmy Lin, and Miles Efron. 2015. Reproducible experiments on lexical and temporal feedback for tweet search. In Proceedings of the 37th European Conference on Information Retrieval (ECIR’15). 755--767.

Phyllis A. Richmond. 1963. Review of the cranfield project. Amer. Document. 14, 4 (1963), 307--311.

Charles Safran, Meryl Bloomrosen, W. Edward Hammond, Steven Labkoff, Suzanne Markel-Fox, Paul C. Tang, and Don E. Detmer. 2007. Toward a national framework for the secondary use of health data: An American medical informatics association white paper. J. Amer. Med. Info. Assoc. 14, 1 (2007), 1--9.

Mario Scriminaci, Andreas Lommatzsch, Benjamin Kille, Frank Hopfgartner, Martha Larson, Davide Malagoli, András Serény, and Till Plumbaum. 2016. Idomaar: A framework for multi-dimensional benchmarking of recommender algorithms. In Proceedings of the Poster Track of the 10th ACM Conference on Recommender Systems (RecSys’16).

Karen Spärck Jones and Cornelius Joost van Rijsbergen. 1975. Report on the Need for and Provision of an Ideal Information Retrieval Test Collection. British Library Research and Development Report 5266. Computer Laboratory, University of Cambridge.

Efstathios Stamatatos, Martin Potthast, Francisco Rangel, Paolo Rosso, and Benno Stein. 2015. Overview of the PAN/CLEF 2015 evaluation lab. In Proceedings of the International Conference of the CLEF Initiative (CLEF’15), Josiane Mothe, Jacques Savoy, Jaap Kamps, Karen Pinel-Sauvagnat, Gareth J.F. Jones, Eric SanJuan, Linda Cappellato, and Nicola Ferro (Eds.). Springer, Berlin, 518--538.

Joseph E. Stiglitz and Scott J. Wallsten. 1999. Public-private technology partnerships: Promises and pitfalls. Amer. Behav. Sci. 43, 1 (1999), 52--73.

Victoria Stodden. 2009. The legal framework for reproducible scientific research: Licensing and copyright. Comput. Sci. Eng. 11, 1 (Feb. 2009), 35—40.

Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. YFCC100M: The new data in multimedia research. Commun. ACM 59, 2 (Jan. 2016), 64--73.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos et al. 2015. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformat. 16, 1 (2015), 138.

Marieke van Erp, Pablo Mendes, Heiko Paulheim, Filip Ilievski, Julien Plu, Giuseppe Rizzo, and Joerg Waitelonis. 2016. Evaluating entity linking: An analysis of current benchmark datasets and a roadmap for doing a better job. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC’16) (23--28), Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association (ELRA), Paris, France.

Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. 2013. OpenML: Networked science in machine learning. SIGKDD Explor. 15, 2 (2013), 49--60.

Ellen M. Voorhees and Donna K. Harman (Eds.). 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT Press.

Richard Watermeyer. 2016. Impact in the REF: Issues and obstacles. Studies Higher Edu. 41, 2 (2016), 199--214.

Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the 19th Conference on Computational Natural Language Learning—Shared Task. Association for Computational Linguistics, 1--16.

Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Attapol Rutherford, Bonnie Webber, Chuan Wang, and Hongmin Wang. 2016. CoNLL 2016 shared task on multilingual shallow discourse parsing. In Proceedings of the CoNLL 2016 Shared Task. Association for Computational Linguistics, 1--19.

Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Václava Kettnerová, Zdenka Uresova, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Çağrı Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics, 1--19.