research-article

Open access

Naiad: a timely dataflow system

Authors:

Derek G. Murray,

Frank McSherry,

Rebecca Isaacs,

Martín AbadiAuthors Info & Claims

SOSP '13: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles

Pages 439 - 455

https://doi.org/10.1145/2517349.2522738

Published: 03 November 2013 Publication History

Abstract

Naiad is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations. Although existing systems offer some of these features, applications that require all three have relied on multiple platforms, at the expense of efficiency, maintainability, and simplicity. Naiad resolves the complexities of combining these features in one framework.

A new computational model, timely dataflow, underlies Naiad and captures opportunities for parallelism across a wide class of algorithms. This model enriches dataflow computation with timestamps that represent logical points in the computation and provide the basis for an efficient, lightweight coordination mechanism.

We show that many powerful high-level programming models can be built on Naiad's low-level primitives, enabling such diverse tasks as streaming data analysis, iterative machine learning, and interactive graph mining. Naiad outperforms specialized systems in their target application domains, and its unique features enable the development of new high-performance applications.

Supplementary Material

MP4 File (d3-05-derek-murray.mp4)

Download
855.60 MB

References

[1]

The ClueWeb09 Dataset. http://lemurproject.org/clueweb09.

[2]

Parallel Data Warehouse. http://www.microsoft.com/en-us/sqlserver/solutions-technologies/data-warehousing/pdw.aspx.

[3]

Storm: Distributed and fault-tolerant realtime computation. http://storm-project.net/.

[4]

M. Abadi, F. McSherry, D. G. Murray, and T. L. Rodeheffer. Formal analysis of a distributed algorithm for tracking progress. In Proceedings of the IFIP Joint International Conference on Formal Techniques for Distributed Systems, June 2013.

[5]

T. Akidau, A. Balikov, K. Bekiroǧlu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. MillWheel: fault-tolerant stream processing at Internet scale. In Proceedings of the 39th International Conference on Very Large Data Bases (VLDB), Aug. 2013.

Digital Library

[6]

M. Alizadeh, A. Greenberg, D. A. Maltz, J. Padhyey, P. Pately, B. Prabhakarz, S. Senguptay, and M. Sridharany. Data Center TCP (DCTCP). In Proceedings of the ACM International Conference on Applications, Technologies, Architectures and Protocols for Computer Communications (SIGCOMM), Aug. 2010.

Digital Library

[7]

P. Alvaro, N. Conway, J. M. Hellerstein, and W. R. Marczak. Consistency analysis in Bloom: a CALM and collected approach. In Proceedings of the 5th Conference on Innovative Data Systems Research (CIDR), Jan. 2011.

[8]

B. Calder, J. Wang, A. Ogus, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J. Haridas, C. Uddaraju, H. Khatri, A. Edwards, V. Bedekar, S. Mainali, R. Abbasi, A. Agarwal, M. F. ul Haq, M. I. ul Haq, D. Bhardwaj, S. Dayanand, A. Adusumilli, M. McNett, S. Sankaran, K. Manivannan, and L. Rigas. Windows Azure Storage: a highly available cloud storage service with strong consistency. In Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP), Oct. 2011.

Digital Library

[9]

B. Chandramouli, J. Goldstein, and D. Maier. On-the-fly progress detection in iterative stream queries. Proceedings of the Very Large Database Endowment (PVLDB), 2(1):241--252, Aug. 2009.

Digital Library

[10]

R. Cheng, J. Hong, A. Kyrola, Y. Miao, X. Weng, M. Wu, F. Yang, L. Zhou, F. Zhao, and E. Chen. Kineograph: taking the pulse of a fast-changing and connected world. In Proceedings of the EuroSys Conference, Apr. 2012.

Digital Library

[11]

J. Cipar, Q. Ho, J. K. Kim, S. Lee, G. R. Ganger, G. Gibson, K. Keeton, and E. Xing. Solving the straggler problem with bounded staleness. In Proceedings of the 14th Workshop on Hot Topics in Operating Systems (HotOS), May 2013.

Digital Library

[12]

D. D. Clark. Window and acknowledgement strategy in TCP. RFC 813, July 1982.

Digital Library

[13]

N. Conway, W. R. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. Logic and lattices for distributed programming. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC), Oct. 2012.

Digital Library

[14]

J. Dean and L. A. Barroso. The tail at scale. Communications of the ACM, 56(2):74--80, Feb. 2013.

Digital Library

[15]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Dec. 2004.

Digital Library

[16]

J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: distributed graph-parallel computation on natural graphs. In Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Oct. 2012.

Digital Library

[17]

D. Hsu, N. Karampatziakis, J. Langford, and A. Smola. Parallel online learning. In R. Bekkerman, M. Bilenko, and J. Langford, editors, Scaling Up Machine Learning: Parallel and Distributed Approaches. Cambridge University Press, Dec. 2011.

[18]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In Proceedings of the EuroSys Conference, Mar. 2007.

Digital Library

[19]

Q. Ke, M. Isard, and Y. Yu. Optimus: A dynamic rewriting framework for execution plans of data-parallel computation. In Proceedings of the EuroSys Conference, Apr. 2013.

Digital Library

[20]

E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. F. Kaashoek. The Click Modular Router. ACM Transactions on Computer Systems, 18(3):263--297, Aug. 2000.

Digital Library

[21]

H. Kwak, C. Lee, H. Park, and S. Moon. What is Twitter, a social network or a news media? In Proceedings of the 19th International World Wide Web Conference (WWW), Apr. 2010.

Digital Library

[22]

J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, and D. Maier. Out-of-order processing: a new architecture for high-performance stream systems. Proceedings of the Very Large Database Endowment (PVLDB), 1(1):274--288, Aug. 2008.

Digital Library

[23]

B. T. Loo, T. Condie, M. Garofalakis, D. E. Gay, J. M. Hellerstein, P. Maniatis, R. Ranakrishnan, T. Roscoe, and I. Stoica. Declarative networking: language, execution and optimization. In Proceedings of the ACM International Conference on Management of Data (SIGMOD), June 2006.

Digital Library

[24]

B. T. Loo, T. Condie, J. M. Hellerstein, P. Maniatis, T. Roscoe, and I. Stoica. Implementing declarative overlays. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP), Oct. 2005.

Digital Library

[25]

B. T. Loo, J. M. Hellerstein, I. Stoica, and R. Ramakrishnan. Declarative routing: extensible routing with declarative queries. In Proceedings of the ACM International Conference on Applications, Technologies, Architectures and Protocols for Computer Communications (SIGCOMM), Aug. 2005.

Digital Library

[26]

Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. GraphLab: A new parallel framework for machine learning. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI), July 2010.

[27]

G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In Proceedings of the ACM International Conference on Management of Data (SIGMOD), June 2010.

Digital Library

[28]

F. McSherry, D. G. Murray, R. Isaacs, and M. Isard. Differential dataflow. In Proceedings of the 6th Conference on Innovative Data Systems Research (CIDR), Jan. 2013.

[29]

C. Mitchell, R. Power, and J. Li. Oolong: asynchronous distributed applications made easy. In Proceedings of the 3rd Asia-Pacific Workshop on Systems (APSys), July 2012.

Digital Library

[30]

D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand. CIEL: a universal execution engine for distributed dataflow computing. In Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI), Mar. 2011.

Digital Library

[31]

D. Nagle, D. Serenyi, and A. Matthews. The Panasas ActiveScale storage cluster: Delivering scalable high bandwidth storage. In Proceedings of the ACM/IEEE Supercomputing Conference (SC), Nov. 2004.

Digital Library

[32]

J. Nagle. Congestion control in IP/TCP internetworks. RFC 896, Jan. 1984.

Digital Library

[33]

M. Najork. The scalable hyperlink store. In Proceedings of the 20th ACM Conference on Hypertext and Hypermedia, June 2009.

Digital Library

[34]

M. Najork, D. Fetterly, A. Halverson, K. Kenthapadi, and S. Gollapudi. Of hammers and nails: an empirical comparison of three paradigms for processing large graphs. In Proceedings of the 5th ACM International Conference on Web Search and Data Mining (WSDM), Feb. 2012.

Digital Library

[35]

J. Pelissier. Providing quality of service over InfiniBand#8482; Architecture fabrics. In Proceedings of the 8th IEEE Symposium on High Performance Interconnects (HOT Interconnects), 2000.

[36]

D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Oct. 2010.

Digital Library

[37]

D. P. Reed and R. K. Kanodia. Synchronization with eventcounts and sequencers. Communications of the ACM, 22(2):115--123, Feb. 1979.

Digital Library

[38]

P. A. Tucker, D. Maier, T. Sheard, and L. Fegaras. Exploiting punctuation semantics in continuous data streams. IEEE Transactions on Knowledge and Data Engineering, 15(3), May/June 2002.

Digital Library

[39]

M. Welsh, D. Culler, and E. Brewer. SEDA: an architecture for well-conditioned, scalable internet services. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP), Oct. 2001.

Digital Library

[40]

R. Xin, J. Gonzalez, M. Franklin, and I. Stoica. GraphX: A resilient distributed graph system on spark. In Proceedings of the Graph Data-management Experiences and Systems (GRADES) Workshop, June 2013.

Digital Library

[41]

Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Dec. 2008.

Digital Library

[42]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient Distributed Datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI), Apr. 2012.

Digital Library

[43]

M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, and I. Stoica. Discretized Streams: Fault-tolerant streaming computation at scale. In Proceedings of the 24th ACM Symposium on Operating Systems Principles (SOSP), Nov. 2013.

Digital Library

[44]

M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving MapReduce performance in heterogeneous environments. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Dec. 2008.

Digital Library

[45]

Y. Zhang, Q. Gao, L. Gao, and C. Wang. PrIter: A distributed framework for prioritized iterative computations. In Proceedings of the 2nd ACM Symposium on Cloud Computing (SoCC), Oct. 2011.

Digital Library

[46]

Y. Zhang, Q. Gao, L. Gao, and C. Wang. Accelerate large-scale iterative computation through asynchronous accumulative updates. In Proceedings of the 3rd ACM Workshop on Scientific Cloud Computing (ScienceCloud), June 2012.

Digital Library

Cited By

Wang ZHuang YNi SKumar AAlsudais SLiu XLin XDing YLi C(2024)Texera: A System for Collaborative and Interactive Data Analytics Using WorkflowsProceedings of the VLDB Endowment10.14778/3681954.368202217:11(3580-3588)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682022
Lyu CFan QGuyard PDiao Y(2024)A Spark Optimizer for Adaptive, Fine-Grained Parameter TuningProceedings of the VLDB Endowment10.14778/3681954.368202117:11(3565-3579)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682021
Guliyev RHaldar AFerhatosmanoglu H(2024)D3-GNN: Dynamic Distributed Dataflow for Streaming Graph Neural NetworksProceedings of the VLDB Endowment10.14778/3681954.368196117:11(2764-2777)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3681961
Show More Cited By

Recommendations

A study of higher-order sequential computation
A robust algorithm for approximate compatible observability don't care (CODC) computation
DAC '04: Proceedings of the 41st annual Design Automation Conference

Compatible Observability Don't Cares (CODCs) are a powerful means to express the flexibility present at a node in a multi-level logic network. Despite their elegance, the applicability of CODCs has been hampered by their computational complexity. The ...
An experimental analysis of self-adjusting computation

Recent work on adaptive functional programming (AFP) developed techniques for writing programs that can respond to modifications to their data by performing change propagation. To achieve this, executions of programs are represented with dynamic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SOSP '13: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles

November 2013

498 pages

ISBN:9781450323888

DOI:10.1145/2517349

General Chair:
Michael Kaminsky
Intel Labs
,
Program Chair:
Mike Dahlin
Google and UT Austin

Copyright © 2013 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2013

Check for updates

Qualifiers

Research-article

Conference

SOSP '13

Sponsor:

SIGOPS

SOSP '13: ACM SIGOPS 24th Symposium on Operating Systems Principles

November 3 - 6, 2013

Pennsylvania, Farminton

Acceptance Rates

Overall Acceptance Rate 131 of 716 submissions, 18%

Upcoming Conference

SOSP '24

Sponsor:
sigops

ACM SIGOPS 30th Symposium on Operating Systems Principles

November 4 - 6, 2024

Austin , TX , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

577
Total Citations
View Citations
22,136
Total Downloads

Downloads (Last 12 months)1,782
Downloads (Last 6 weeks)183

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZHuang YNi SKumar AAlsudais SLiu XLin XDing YLi C(2024)Texera: A System for Collaborative and Interactive Data Analytics Using WorkflowsProceedings of the VLDB Endowment10.14778/3681954.368202217:11(3580-3588)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682022
Lyu CFan QGuyard PDiao Y(2024)A Spark Optimizer for Adaptive, Fine-Grained Parameter TuningProceedings of the VLDB Endowment10.14778/3681954.368202117:11(3565-3579)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3682021
Guliyev RHaldar AFerhatosmanoglu H(2024)D3-GNN: Dynamic Distributed Dataflow for Streaming Graph Neural NetworksProceedings of the VLDB Endowment10.14778/3681954.368196117:11(2764-2777)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.14778/3681954.3681961
Wheatman BDong XShen ZDhulipala LŁącki JPandey PXu H(2024)BYO: A Unified Framework for Benchmarking Large-Scale Graph ContainersProceedings of the VLDB Endowment10.14778/3665844.366585917:9(2307-2320)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.14778/3665844.3665859
Nguyen DNguyen K(2024)Polar: A Managed Runtime with Hotness-Segregated Heap for Far MemoryProceedings of the 15th ACM SIGOPS Asia-Pacific Workshop on Systems10.1145/3678015.3680490(15-22)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3678015.3680490
Lin SYang QYang ZWang YZhao S(2024)LubeRDMA: A Fail-safe Mechanism of RDMAProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3663411(16-22)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3663411
Sahu SSalihoglu S(2024)Optimizing Differential Computation for Large-Scale Graph ProcessingProceedings of the 7th Joint Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA)10.1145/3661304.3661900(1-9)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3661304.3661900
Sharma AMehta DWu B(2024)Understanding High-Performance Subgraph Pattern Matching: A Systems PerspectiveProceedings of the 7th Joint Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA)10.1145/3661304.3661897(1-12)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3661304.3661897
Zakhour GWeisenburger PSalvaneschi G(2024)Automated Verification of Fundamental Algebraic LawsProceedings of the ACM on Programming Languages10.1145/36564088:PLDI(766-789)Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1145/3656408
Kozar ADel Monte BZeuch SMarkl V(2024)Fault Tolerance Placement in the Internet of ThingsProceedings of the ACM on Management of Data10.1145/36549412:3(1-29)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3654941
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents