demonstration

Shark: fast data analysis using coarse-grained distributed memory

Authors:

Ion StoicaAuthors Info & Claims

SIGMOD '12: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

Pages 689 - 692

https://doi.org/10.1145/2213836.2213934

Published: 20 May 2012 Publication History

Get Access

Abstract

Shark is a research data analysis system built on a novel coarse-grained distributed shared-memory abstraction. Shark marries query processing with deep data analysis, providing a unified system for easy data manipulation using SQL and pushing sophisticated analysis closer to data. It scales to thousands of nodes in a fault-tolerant manner. Shark can answer queries 40X faster than Apache Hive and run machine learning programs 25X faster than MapReduce programs in Apache Hadoop on large datasets.

References

[1]

G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. Disk-locality in datacenter computing considered irrelevant. In HotOS '11, 2011.

Digital Library

Google Scholar

[2]

A. Pavlo, E. Paulson, A. Rasin, D. Abadi, D. DeWitt, S. Madden, and M. Stonebraker. A comparison of approaches to large-scale data analysis. In Proceedings of the 35th SIGMOD international conference on Management of data, pages 165--178. ACM, 2009.

Digital Library

Google Scholar

[3]

A. Thusoo, J. Sarma, N. Jain, Z. Shao, P. Chakka, N. Zhang, S. Antony, H. Liu, and R. Murthy. Hive-a petabyte scale data warehouse using hadoop. In Data Engineering (ICDE), 2010 IEEE 26th International Conference on, pages 996--1005. IEEE, 2010.

Crossref

Google Scholar

[4]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI 2012.

Digital Library

Google Scholar

Cited By

View all

Patidar SKumar NJindal R(2024)IoT Data Stream Handling, Analysis, Communication and Security Issues: A Systematic SurveyWireless Personal Communications10.1007/s11277-024-11177-1Online publication date: 20-May-2024
https://doi.org/10.1007/s11277-024-11177-1
Mandal P(2023)Marketing Information and Marketing IntelligenceInternational Journal of Business Strategy and Automation10.4018/IJBSA.3162353:1(1-12)Online publication date: 13-Jan-2023
https://doi.org/10.4018/IJBSA.316235
Mandal P(2022)Marketing Information and Marketing IntelligenceInternational Journal of Technology Diffusion10.4018/IJTD.30074513:1(1-14)Online publication date: 20-May-2022
https://dl.acm.org/doi/10.4018/IJTD.300745
Show More Cited By

Index Terms

Shark: fast data analysis using coarse-grained distributed memory
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs

Recommendations

Spark SQL: Relational Data Processing in Spark
SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

Spark SQL is a new module in Apache Spark that integrates relational processing with Spark's functional programming API. Built on our experience with Shark, Spark SQL lets Spark programmers leverage the benefits of relational processing (e.g. ...
Shark: SQL and rich analytics at scale
SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data

Shark is a new data analysis system that marries query processing with complex analytics on large clusters. It leverages a novel distributed memory abstraction to provide a unified engine that can run SQL queries and sophisticated analytics functions (...
Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware
IDEAS '17: Proceedings of the 21st International Database Engineering & Applications Symposium

Big Data is currently conceptualized as data whose volume, variety or velocity impose significant difficulties in traditional techniques and technologies. Big Data Warehousing is emerging as a new concept for Big Data analytics. In this context, SQL-on-...

Comments

Information & Contributors

Information

Published In

SIGMOD '12: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data

May 2012

886 pages

ISBN:9781450312479

DOI:10.1145/2213836

General Chairs:
K. Selçuk Candan
Arizona State University
,
Yi Chen
Arizona State University
,
Richard Snodgrass
University of Arizona
,
Program Chair:
Luis Gravano
Columbia University
,
Publications Chair:
Ariel Fuxman
Microsoft Research

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 May 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Demonstration

Conference

SIGMOD/PODS '12

Sponsor:

SIGMOD

SIGMOD/PODS '12: International Conference on Management of Data

May 20 - 24, 2012

Arizona, Scottsdale, USA

Acceptance Rates

SIGMOD '12 Paper Acceptance Rate 48 of 289 submissions, 17%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

86
Total Citations
View Citations
1,561
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to

Other Metrics

View Author Metrics

Citations

Cited By

View all

Patidar SKumar NJindal R(2024)IoT Data Stream Handling, Analysis, Communication and Security Issues: A Systematic SurveyWireless Personal Communications10.1007/s11277-024-11177-1Online publication date: 20-May-2024
https://doi.org/10.1007/s11277-024-11177-1
Mandal P(2023)Marketing Information and Marketing IntelligenceInternational Journal of Business Strategy and Automation10.4018/IJBSA.3162353:1(1-12)Online publication date: 13-Jan-2023
https://doi.org/10.4018/IJBSA.316235
Mandal P(2022)Marketing Information and Marketing IntelligenceInternational Journal of Technology Diffusion10.4018/IJTD.30074513:1(1-14)Online publication date: 20-May-2022
https://dl.acm.org/doi/10.4018/IJTD.300745
Jayender PKundu G(2022)IoT Analytics and ERP Interoperability in Automotive SCMInternational Journal of Fuzzy System Applications10.4018/IJFSA.30628211:3(1-19)Online publication date: 29-Jul-2022
https://dl.acm.org/doi/10.4018/IJFSA.306282
Usman SMehmood RKatib IAlbeshri A(2022)Data Locality in High Performance Computing, Big Data, and Converged Systems: An Analysis of the Cutting Edge and a Future System ArchitectureElectronics10.3390/electronics1201005312:1(53)Online publication date: 23-Dec-2022
https://doi.org/10.3390/electronics12010053
Jie HDehua W(2022)Design and Implementation of Automatic Selection of the Most Efficient Itemset Algorithm Based on SparkScientific Programming10.1155/2022/33622032022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/3362203
Patil RDivekar SDahiphale AAttarde GChavan A(2022)Recommender System: Analysing Products Using Data Stream Mining2022 6th International Conference On Computing, Communication, Control And Automation (ICCUBEA10.1109/ICCUBEA54992.2022.10010897(1-4)Online publication date: 26-Aug-2022
https://doi.org/10.1109/ICCUBEA54992.2022.10010897
Pratap C(2021)Online Marketing Research - Roles in Generating Customer InsightsStudies in Business and Economics10.2478/sbe-2021-001216:1(147-161)Online publication date: 26-May-2021
https://doi.org/10.2478/sbe-2021-0012
Liu JWang KChen F(2021)TSCacheProceedings of the VLDB Endowment10.14778/3484224.348422514:13(3253-3266)Online publication date: 28-Oct-2021
https://dl.acm.org/doi/10.14778/3484224.3484225
Zhao YCalheiros RGange GBailey JSinnott R(2021)SLA-Based Profit Optimization Resource Scheduling for Big Data Analytics-as-a-Service Platforms in Cloud Computing EnvironmentsIEEE Transactions on Cloud Computing10.1109/TCC.2018.28899569:3(1236-1253)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TCC.2018.2889956
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Spark SQL: Relational Data Processing in Spark

Shark: SQL and rich analytics at scale

Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Spark SQL: Relational Data Processing in Spark

Shark: SQL and rich analytics at scale

Evaluating SQL-on-Hadoop for Big Data Warehousing on Not-So-Good Hardware

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations