research-article

Realtime Data Processing at Facebook

Authors:

Guoqiang Jerry Chen,

Janet L. Wiener,

Anshul Jaiswal,

Tim Williamson,

Serhat YilmazAuthors Info & Claims

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

Pages 1087 - 1098

https://doi.org/10.1145/2882903.2904441

Published: 14 June 2016 Publication History

Abstract

Realtime data processing powers many use cases at Facebook, including realtime reporting of the aggregated, anonymized voice of Facebook users, analytics for mobile applications, and insights for Facebook page administrators. Many companies have developed their own systems; we have a realtime data processing ecosystem at Facebook that handles hundreds of Gigabytes per second across hundreds of data pipelines.

Many decisions must be made while designing a realtime stream processing system. In this paper, we identify five important design decisions that affect their ease of use, performance, fault tolerance, scalability, and correctness. We compare the alternative choices for each decision and contrast what we built at Facebook to other published systems.

Our main decision was targeting seconds of latency, not milliseconds. Seconds is fast enough for all of the use cases we support and it allows us to use a persistent message bus for data transport. This data transport mechanism then paved the way for fault tolerance, scalability, and multiple options for correctness in our stream processing systems Puma, Swift, and Stylus.

We then illustrate how our decisions and systems satisfy our requirements for multiple use cases at Facebook. Finally, we reflect on the lessons we learned as we built and operated these systems.

References

[1]

Monoid. https://en.wikipedia.org/wiki/Monoid.

[2]

Presto. http://prestodb.io.

[3]

Rocksdb. http://rocksdb.org.

[4]

Samza. http://samza.apache.org.

[5]

Scribe. https://github.com/facebook/scribe.

[6]

Zeromq. https://zeromq.org/.

[7]

L. Abraham, J. Allen, O. Barykin, V. Borkar, B. Chopra, C. Gerea, D. Merl, J. Metzler, D. Reiss, S. Subramanian, et al. Scuba: diving into data at facebook. In PVLDB, pages 1057--1067, 2013.

Digital Library

[8]

A. Agarwal, M. Slee, and M. Kwiatkowski. Thrift: Scalable cross-language services implementation. Technical report, Facebook, 2007.

[9]

T. Akidau, A. Balikov, K. Bekiro\uglu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. Millwheel: Fault-tolerant stream processing at internet scale. PVLDB, 6(11):1033--1044, Aug. 2013.

Digital Library

[10]

T. Akidau, R. Bradshaw, C. Chambers, S. Chernyak, R. J. Fernández-Moctezuma, R. Lax, S. McVeety, D. Mills, F. Perry, E. Schmidt, and S. Whittle. The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. PVLDB, 8(12):1792--1803, Aug. 2015.

Digital Library

[11]

R. Ananthanarayanan, V. Basker, S. Das, A. Gupta, H. Jiang, T. Qiu, A. Reznichenko, D. Ryabkov, M. Singh, and S. Venkataraman. Photon: fault-tolerant and scalable joining of continuous data streams. In SIGMOD, pages 577--588, 2013.

Digital Library

[12]

A. Arasu, B. Babcock, S. Babu, J. Cieslewicz, M. Datar, K. Ito, R. Motwani, U. Srivastava, and J. Widom. Stream: The stanford data stream management system. Technical Report 2004--20, Stanford InfoLab, 2004.

[13]

M. Balazinska, H. Balakrishnan, S. R. Madden, and M. Stonebraker. Fault-tolerance in the borealis distributed stream processing system. ACM TODS, 33(1):3:1--3:44, Mar. 2008.

Digital Library

[14]

O. Boykin, S. Ritchie, I. O'Connell, and J. Lin. Summingbird: A framework for integrating batch and online mapreduce computations. PVLDB, 7(13):1441--1451, Aug. 2014.

Digital Library

[15]

N. Bronson, T. Lento, and J. L. Wiener. Open data challenges at facebook. In ICDE, pages 1516--1519, 2015.

[16]

P. Carbone, G. Fóra, S. Ewen, S. Haridi, and K. Tzoumas. Lightweight asynchronous snapshots for distributed dataflows. CoRR, abs/1506.08603, 2015.

[17]

R. Castro Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management. In SIGMOD, pages 725--736, 2013.

Digital Library

[18]

A. Goel, B. Chopra, C. Gerea, D. Mátáni, J. Metzler, F. Ul Haq, and J. L. Wiener. Fast database restarts at Facebook. In SIGMOD, pages 541--549, 2014.

Digital Library

[19]

J. Kreps, N. Narkhede, and J. Rao. Kafka: A distributed messaging system for log processing. In SIGMOD Workshop on Networking Meets Databases, 2011.

[20]

S. Kulkarni, N. Bhagat, M. Fu, V. Kedigehalli, C. Kellogg, S. Mittal, J. M. Patel, K. Ramasamy, and S. Taneja. Twitter heron: Stream processing at scale. In SIGMOD, pages 239--250, 2015.

Digital Library

[21]

J. Meehan, N. Tatbul, S. Zdonik, C. Aslantas, U. Cetintemel, J. Du, T. Kraska, S. Madden, D. Maier, A. Pavlo, M. Stonebraker, K. Tufte, and H. Wang. S-store: Streaming meets transaction processing. PVLDB, 8(13):2134--2145, Sept. 2015.

Digital Library

[22]

L. Neumeyer, B. Robbins, A. Nair, and A. Kesari. S4: Distributed stream computing platform. In IEEE Data Mining Workshops, pages 170--177, 2010.

Digital Library

[23]

K. Shvachko, H. Kuang, S. Radia, and R. Chansler. The Hadoop distributed file system. In Mass Storage Systems and Technologies (MSST), pages 1--10, 2010.

Digital Library

[24]

M. Stonebraker, U. Çetintemel, and S. B. Zdonik. The 8 requirements of real-time stream processing. SIGMOD Record, 34(4):42--47, 2005.

Digital Library

[25]

J. Sundram. Developing data products. Big Data Spain, 2015. https://www.youtube.com/watch?v=CkEdD6FL7Ug.

[26]

A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen Sarma, R. Murthy, and H. Liu. Data warehousing and analytics infrastructure at facebook. In SIGMOD, pages 1013--1020, 2010.

Digital Library

[27]

R. Tibbetts, S. Yang, R. MacNeill, and D. Rydzewski. Streambase liveview: Push-based real-time analytics. StreamBase Systems (Jan 2012), 2011.

[28]

A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, N. Bhagat, S. Mittal, and D. Ryaboy. Storm@twitter. In SIGMOD, pages 147--156, 2014.

Digital Library

[29]

J. L. Wiener. Understanding realtime conversations on Facebook. QCON San Francisco, 2015. https://qconsf.com/sf2015/speakers/janet-wiener.

[30]

Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda, and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. In OSDI, volume 8, pages 1--14, 2008.

Digital Library

[31]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In NSDI, pages 2--2, 2012.

Digital Library

[32]

M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica. Discretized streams: Fault-tolerant streaming computation at scale. In SOSP, pages 423--438, 2013.

Digital Library

Cited By

Xiao YZhao YLin SKuzmanovic A(2024)Snatch: Online Streaming Analytics at the Network EdgeProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629577(349-369)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3629577
Mishra S(2024)A survey of LSM-Tree based Indexes, Data Systems and KV-stores2024 IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS)10.1109/SCEECS61402.2024.10482249(1-6)Online publication date: 24-Feb-2024
https://doi.org/10.1109/SCEECS61402.2024.10482249
Zhang JWang FQiu SWang YOu JHuang JLi BFang PFeng D(2024)Scavenger: Better Space-Time Trade-Offs for Key-Value Separated LSM-trees2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00312(4072-4085)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00312
Show More Cited By

Recommendations

Dual-Paradigm Stream Processing
ICPP '18: Proceedings of the 47th International Conference on Parallel Processing

Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable ...
Scalable stateful stream processing for smart grids
DEBS '14: Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems

We describe a solution to the ACM DEBS Grand Challenge 2014, which evaluates event-based systems for smart grid analytics. Our solution follows the paradigm of stateful data stream processing and is implemented on top of the SEEP stream processing ...
A New Application Benchmark for Data Stream Processing Architectures in an Enterprise Context: Doctoral Symposium
DEBS '17: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems

Against the backdrop of ever-growing data volumes and trends like the Internet of Things (IoT) or Industry 4.0, Data Stream Processing Systems (DSPSs) or data stream processing architectures in general receive a greater interest. Continuously analyzing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMOD '16: Proceedings of the 2016 International Conference on Management of Data

June 2016

2300 pages

ISBN:9781450335317

DOI:10.1145/2882903

General Chairs:
Fatma Özcan
IBM Research, USA
,
Georgia Koutrika
HP Labs, USA
,
Program Chair:
Sam Madden
Massachusetts Institute of Technology, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS'16

Sponsor:

SIGMOD

SIGMOD/PODS'16: International Conference on Management of Data

June 26 - July 1, 2016

California, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 785 of 4,003 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

87
Total Citations
View Citations
2,602
Total Downloads

Downloads (Last 12 months)150
Downloads (Last 6 weeks)9

Reflects downloads up to 11 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xiao YZhao YLin SKuzmanovic A(2024)Snatch: Online Streaming Analytics at the Network EdgeProceedings of the Nineteenth European Conference on Computer Systems10.1145/3627703.3629577(349-369)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1145/3627703.3629577
Mishra S(2024)A survey of LSM-Tree based Indexes, Data Systems and KV-stores2024 IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS)10.1109/SCEECS61402.2024.10482249(1-6)Online publication date: 24-Feb-2024
https://doi.org/10.1109/SCEECS61402.2024.10482249
Zhang JWang FQiu SWang YOu JHuang JLi BFang PFeng D(2024)Scavenger: Better Space-Time Trade-Offs for Key-Value Separated LSM-trees2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00312(4072-4085)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00312
Qiu LChen LJie HKe XGao YLiu YZhang Z(2024)GPU-Accelerated Batch-Dynamic Subgraph Matching2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00248(3204-3216)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00248
Vergilio TKor AMullier D(2023)A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud EnvironmentApplied Sciences10.3390/app13231263513:23(12635)Online publication date: 23-Nov-2023
https://doi.org/10.3390/app132312635
Mao YChen ZZhang YWang MFang YZhang GShi RMa R(2023)StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDanceProceedings of the VLDB Endowment10.14778/3611540.361154316:12(3501-3514)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.14778/3611540.3611543
Xu HLi AWheatman BMarneni MPandey P(2023)BP-Tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-TreesProceedings of the VLDB Endowment10.14778/3611479.361150216:11(2976-2989)Online publication date: 24-Aug-2023
https://dl.acm.org/doi/10.14778/3611479.3611502
Sahraei ADemetriou SSobhgol AZhang HNagaraja APathak NJoshi GSouza CHuang BCook WGolovei AVenkat PMcfague ASkarlatos DPatel VThind RGonzalez EJin YTang CDruschel PKaufmann AMace JFlinn JSeltzer M(2023)XFaaS: Hyperscale and Low Cost Serverless Functions at MetaProceedings of the 29th Symposium on Operating Systems Principles10.1145/3600006.3613155(231-246)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1145/3600006.3613155
Wu NVatsalan DKaafar MRamesh S(2023)Privacy-Preserving Record Linkage for Cardinality CountingProceedings of the 2023 ACM Asia Conference on Computer and Communications Security10.1145/3579856.3590338(53-64)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3579856.3590338
Song YKim WMonga SMin CEom YAamodt TJerger NSwift M(2023)Prism: Optimizing Key-Value Store for Modern Heterogeneous Storage DevicesProceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 210.1145/3575693.3575722(588-602)Online publication date: 27-Jan-2023
https://dl.acm.org/doi/10.1145/3575693.3575722
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents