Information Retrieval Meets Scalable Text Analytics: Solr Integration with Spark
Pages 1313 - 1316
Abstract
Despite the broad adoption of both Apache Spark and Apache Solr, there is little integration between these two platforms to support scalable, end-to-end text analytics. We believe this is a missed opportunity, as there is substantial synergy in building analytical pipelines where the results of potentially complex faceted queries feed downstream text processing components. This demonstration explores exactly such an integration: we evaluate performance under different analytical scenarios and present three simple case studies that illustrate the range of possible analyses enabled by seamlessly connecting Spark to Solr.
References
[1]
G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. 2011. Disk-Locality in Datacenter Computing Considered Irrelevant. In HotOS.
[2]
R. Clancy, T. Eskildsen, N. Ruest, and J. Lin. 2019. Solr Integration in the Anserini Information Retrieval Toolkit. In SIGIR.
[3]
M. Efron, J. Lin, J. He, and A. de Vries. 2014. Temporal Feedback for Tweet Search with Non-Parametric Density Estimation. In SIGIR. 33--42.
[4]
T. Elsayed, F. Ture, and J. Lin. 2010. Brute-Force Approaches to Batch Retrieval: Scalable Indexing with MapReduce, or Why Bother? Technical Report HCIL-2010-23. University of Maryland, College Park, Maryland.
[5]
S. Hendrickson, S. Sturdevant, T. Harter, V. Venkataramani, A. Arpaci-Dusseau, and R. Arpaci-Dusseau. 2016. Serverless Computation with OpenLambda. In HotCloud.
[6]
D. Hiemstra and C. Hauff. 2010. MapReduce for Information Retrieval Evaluation: "Let's Quickly Test This on 12 TB of Data". In CLEF. 64--69.
[7]
Y. Hu. 2005. Efficient, High-Quality Force-Directed Graph Drawing. Mathematica Journal, Vol. 10, 1 (2005), 37--71.
[8]
J. Lin and M. Efron. 2013. Overview of the TREC-2013 Microblog Track. In TREC.
[9]
J. Lin, D. Ryaboy, and K. Weil. 2011. Full-Text Indexing for Optimizing Selection Operations in Large-Scale Data Analytics. In MAPREDUCE. 59--66.
[10]
C. Macdonald. 2018. Combining Terrier with Apache Spark to Create Agile Experimental Information Retrieval Pipelines. In SIGIR. 1309--1312.
[11]
C. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. Bethard, and D. McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In ACL Demos. 55--60.
[12]
F. Moretti. 2013. Distant Reading.
[13]
P. Yang, H. Fang, and J. Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. In SIGIR. 1253--1256.
[14]
P. Yang, H. Fang, and J. Lin. 2018. Anserini: Reproducible Ranking Baselines Using Lucene. JDIQ, Vol. 10, 4 (2018), Article 16.
Index Terms
- Information Retrieval Meets Scalable Text Analytics: Solr Integration with Spark
Recommendations
A knowledge-driven approach for designing data analytics platforms
AbstractBig data analytics technologies are rapidly expanding across all industry sectors as organisations try to make analytics an integral part of their everyday decision-making. Although there are many software tools and libraries to assist analysts ...
WSDM'15 Workshop Summary / Scalable Data Analytics: Theory and Applications
WSDM '15: Proceedings of the Eighth ACM International Conference on Web Search and Data MiningThe SDA workshop at WSDM 2015 is the fifth International Workshop on Scalable Data Analytics, following the previous four workshops of SDA respectively held at IEEE Big Data 2013, PAKDD 2014, IEEE Big Data 2014, and IEEE ICDM 2014. This series of ...
Scalable machine-learning algorithms for big data analytics: a comprehensive review
Big data analytics is one of the emerging technologies as it promises to provide better insights from huge and heterogeneous data. Big data analytics involves selecting the suitable big data storage and computational framework augmented by scalable ...
Comments
Information & Contributors
Information
Published In
July 2019
1512 pages
ISBN:9781450361729
DOI:10.1145/3331184
- General Chairs:
- Benjamin Piwowarski,
- Max Chevalier,
- Eric Gaussier,
- Program Chairs:
- Yoelle Maarek,
- Jian-Yun Nie,
- Falk Scholer
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 July 2019
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Ontario Research Fund
- Canada Foundation for Innovation Leaders Fund
- Natural Sciences and Engineering Research Council of Canada
Conference
SIGIR '19
Sponsor:
SIGIR '19: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
July 21 - 25, 2019
Paris, France
Acceptance Rates
SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 209Total Downloads
- Downloads (Last 12 months)11
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in