Modern users demand analytical features on fresh, real time data. Offering these analytical features to hundreds of millions of users is a relevant problem encountered by many large scale web companies.

Relational databases and key-value stores can be scaled to provide point lookups for a large number of users but fall apart at the combination of high ingest rates, high query rates at low latency for analytical queries. Online analytical databases typically rely on bulk data loads and are not typically built to handle nonstop operation in demanding web environments. Offline analytical systems have high throughput but do not offer low query latencies nor can scale to serving tens of thousands of queries per second.

We present Pinot, a single system used in production at Linkedin that can serve tens of thousands of analytical queries per second, offers near-realtime data ingestion from streaming data sources, and handles the operational requirements of large web properties. We also provide a performance comparison with Druid, a system similar to Pinot.

References

[1]

Daniel Abadi, Samuel Madden, and Miguel Ferreira. 2006. Integrating compression and execution in column-oriented database systems Proceedings of the 2006 ACM SIGMOD international conference on Management of data. ACM, 671--682.

Digital Library

Google Scholar

[2]

Kevin Beyer and Raghu Ramakrishnan. 1999. Bottom-up computation of sparse and iceberg cube. In ACM SIGMOD Record, Vol. Vol. 28. ACM, 359--370.

Digital Library

Google Scholar

[3]

MKABV Bittorf, Taras Bobrovytsky, Casey Ching Alan Choi Justin Erickson, Martin Grund Daniel Hecht, Matthew Jacobs Ishaan Joshi Lenni Kuff, Dileep Kumar Alex Leblang, Nong Li Ippokratis Pandis Henry Robinson, David Rorke Silvius Rus, John Russell Dimitris Tsirogiannis Skye Wanderman, and Milne Michael Yoder. 2015. Impala: A modern, open-source SQL engine for Hadoop Proceedings of the 7th Biennial Conference on Innovative Data Systems Research.

Google Scholar

[4]

Peter A Boncz, Marcin Zukowski, and Niels Nes. 2005. MonetDB/X100: Hyper-Pipelining Query Execution. CIDR, Vol. Vol. 5. 225--237.

Google Scholar

[5]

Samy Chambi, Daniel Lemire, Robert Godin, Kamel Boukhalfa, Charles R Allen, and Fangjin Yang. 2016 a. Optimizing druid with roaring bitmaps. In Proceedings of the 20th International Database Engineering & Applications Symposium. ACM, 77--86.

Digital Library

Google Scholar

[6]

Samy Chambi, Daniel Lemire, Owen Kaser, and Robert Godin. 2016 b. Better bitmap performance with roaring bitmaps. Software: practice and experience Vol. 46, 5 (2016), 709--719.

Digital Library

Google Scholar

[7]

C. Chen. 2005. Top 10 unsolved information visualization problems. Computer Graphics and Applications, IEEE Vol. 25, 4 (july-aug. 2005), 12--16.

Digital Library

Google Scholar

Cited By

View all

Schulze RSchreiber TYatsishin IDahimene RMilovidov A(2024)ClickHouse - Lightning Fast Analytics for EveryoneProceedings of the VLDB Endowment10.14778/3685800.368580217:12(3731-3744)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685802
Meldrum MCarbone P(2024)μWheel: Aggregate Management for Streams and QueriesProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666031(54-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666031
Chen JShi RChen HZhang LLi RDing WFan LWang HXiong MChen YDong BGuo KLin YLiu XShi HWang PWang ZYang YZhao JZhou DZuo ZLiang Y(2023)Krypton: Real-Time Serving and Analytical SQL Engine at ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154516:12(3528-3542)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611545
Show More Cited By

Index Terms

Pinot: Realtime OLAP for 530 Million Users
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Parallel and distributed DBMSs
        Relational parallel and distributed DBMSs
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Cloud computing

Recommendations

Real-time analytical processing with SQL server
Proceedings of the 41st International Conference on Very Large Data Bases, Kohala Coast, Hawaii

Over the last two releases SQL Server has integrated two specialized engines into the core system: the Apollo column store engine for analytical workloads and the Hekaton in-memory engine for high-performance OLTP workloads. There is an increasing demand ...
Efficient Storage and Querying of Horizontal Tables Using a PIVOT Operation in Commercial Relational DBMSs

In recent years, a horizontal table with a large number of attributes is widely used in OLAP or e-business applications to analyze multidimensional data efficiently. For efficient storing and querying of horizontal tables, recent works have tried to ...
OLAP and NoSQL: Happily Ever After
Advances in Databases and Information Systems
Abstract
NoSQL databases are preferred to relational ones for storing heterogeneous data with variable schema and structure. However, their schemaless nature adds complexity to analytical applications, in which a single OLAP analysis often involves large ...

Comments

Information & Contributors

Information

Published In

SIGMOD '18: Proceedings of the 2018 International Conference on Management of Data

May 2018

1874 pages

ISBN:9781450347037

DOI:10.1145/3183713

General Chairs:
Gautam Das
University of Texas at Arlington, USA
,
Christopher Jermaine
Rice University, USA
,
Philip Bernstein
Microsoft Research, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '18

Sponsor:

SIGMOD

SIGMOD/PODS '18: International Conference on Management of Data

June 10 - 15, 2018

TX, Houston, USA

Acceptance Rates

SIGMOD '18 Paper Acceptance Rate 90 of 461 submissions, 20%;

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
1,087
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)11

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Schulze RSchreiber TYatsishin IDahimene RMilovidov A(2024)ClickHouse - Lightning Fast Analytics for EveryoneProceedings of the VLDB Endowment10.14778/3685800.368580217:12(3731-3744)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685802
Meldrum MCarbone P(2024)μWheel: Aggregate Management for Streams and QueriesProceedings of the 18th ACM International Conference on Distributed and Event-based Systems10.1145/3629104.3666031(54-65)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3629104.3666031
Chen JShi RChen HZhang LLi RDing WFan LWang HXiong MChen YDong BGuo KLin YLiu XShi HWang PWang ZYang YZhao JZhou DZuo ZLiang Y(2023)Krypton: Real-Time Serving and Analytical SQL Engine at ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154516:12(3528-3542)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611545
Harizopoulos SHopper TMo MChandrasekaran SChen TCui YGanesh NHelmling GPham HWong S(2022)Meta's next-generation realtime monitoring and analytics platformProceedings of the VLDB Endowment10.14778/3554821.355484115:12(3522-3534)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.14778/3554821.3554841
Javadi SGandhi A(2022)User-Centric Interference-Aware Load Balancing for Cloud-Deployed ApplicationsIEEE Transactions on Cloud Computing10.1109/TCC.2019.294356010:1(736-748)Online publication date: 1-Jan-2022
https://doi.org/10.1109/TCC.2019.2943560
Luo ZNiu LKorukanti VSun YBasmanova MHe YWang BAgrawal DLuo HTang CSingh ALi YDu PBaliga GFu M(2022)From Batch Processing to Real Time Analytics: Running Presto® at Scale2022 IEEE 38th International Conference on Data Engineering (ICDE)10.1109/ICDE53745.2022.00165(1598-1609)Online publication date: May-2022
https://doi.org/10.1109/ICDE53745.2022.00165
Zeydan EMangues-Bafalluy J(2022)Recent Advances in Data Engineering for NetworkingIEEE Access10.1109/ACCESS.2022.316286310(34449-34496)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3162863
Genz FHufeld CKranzlmüller D(2022)Processing Physiological Sensor Data in Near Real-Time as Social Signals for Their Use on Social Virtual Reality PlatformsExtended Reality10.1007/978-3-031-15553-6_4(44-62)Online publication date: 28-Aug-2022
https://doi.org/10.1007/978-3-031-15553-6_4
Cao WFeng XLiang BZhang TGao YZhang YLi FLi GLi ZIdreos SSrivastava D(2021)LogStoreProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457565(2464-2476)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457565
Fu YSoman CLi GLi ZIdreos SSrivastava D(2021)Real-time Data Infrastructure at UberProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3457552(2503-2516)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3457552
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Real-time analytical processing with SQL server

Efficient Storage and Querying of Horizontal Tables Using a PIVOT Operation in Commercial Relational DBMSs

OLAP and NoSQL: Happily Ever After

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations