research-article

DYCE: A Resilient Shared Memory Paradigm for Heterogenous Distributed Systems without Memory Coherence

Authors:

Ulrich Finkler,

Hubertus Franke,

David S. KungAuthors Info & Claims

CF'17: Proceedings of the Computing Frontiers Conference

Pages 17 - 26

https://doi.org/10.1145/3075564.3075579

Published: 15 May 2017 Publication History

Abstract

Parallel programming paradigms are commonly characterized by the core metrics of scalability, memory use, ease of use, hardware requirements and resiliency. Increasingly the support of heterogeneous environments, for example a mix of CPUs and accelerators, are of interest.

Analysis of the semantics of different classes of parallel programming paradigms and their cost leads to DYCE (Distributed Yet Common Environment), a shared memory, rich but hardware friendly, race and deadlock free parallel programming paradigm that allows for resiliency without the need for explicit check-pointing code. Pointer based structures that span the memory of multiple heterogeneous compute devices are possible. Importantly, data exchange is independent of the specific data structures and does not require serialization and deserialization code, even for data structures such as a dynamic linked radix tree of strings.

The analysis shows that DYCE does not require coherence from the system and thus can be executed with near minimal overhead and hardware requirements, including the page table cost for large unified address spaces that span many devices. We demonstrate efficacy with a prototype.

References

[1]

2008. IEEE Std 1003.1-2008. (2008).

Digital Library

[2]

2010. SPARK. (2010). spark.apache.org

[3]

2012. Conference on Natural Language Learning. (2012).

[4]

J Diaz, C Munoz-Caro, and A Nin. 2012. A Survey of Parallel Programming Models and Tools in the Multi and Many-Core Era. IEEE transactions on parallel and distributed systems, Vol. 23, No. 8 (2012).

Digital Library

[5]

E.W. Dijkstra. 1971. Hierarchical ordering of sequential processes. Acta Informatica (1971).

Digital Library

[6]

J.M. Eisner. 2012. Three New Proabilistic Models for Dependency Parsing: An Exploration. In Proceedings of the 16th International Conference on Computational Linguistics.

Digital Library

[7]

J.L. Hennessy and A. Patterson. 2006. Computer architecture, a quantitative approach. Morgan Kaufmann.

Digital Library

[8]

M. Herlihy and J. Moss. 1993. Transactional memory: Architectural support for lock-free data structures. In Proceedings of the 20th International Symposium on Computer Architecture. 289--300.

Digital Library

[9]

M. Hirzel, H. Andrade, B. Gedik, G. Jacques-Silva, R. Khandekar, V. Kumar, M. Mendell, H. Nasgaard, S. Schneider, R. Soule, and K.-L. Wu. 2013. IBM Streams Processing Language: Analyzing Big Data in motion. IBM Journal of Research and Development, Issue 3/4 (2013).

Digital Library

[10]

Pete Keleher, Alan L. Cox, and Willy Zwaenepoel. 1992. Lazy release consistency for software distributed shared memory. In Proceedings of the 19th annual international symposium on Computer architecture.

Digital Library

[11]

L. Lamport. 1979. How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE Trans. Comput. (1979).

Digital Library

[12]

Xuezhe Ma and Hai Zhao. 2012. Fourth-Order Dependency Parsing. In Proceedings of COLING. 785796.

[13]

Oliver A McBryan. 1994. An overview of message passing environments. Parallel Computing Volume 20, Issue 4 (1994). 417--444.

Digital Library

[14]

R. McDonald, K. Crammer, and F. Pereira. 2005. Online Large-Margin Training of Dependency Parsers. In Proceedings ACL.

Digital Library

[15]

Jacob Nelson, Brandon Holt, Brandon Myers, Preston Brigg, Luis Ceze, Simon Kahan, and Mark Oskin. 2015. Latency-Tolerant Software Distributed Shared Memory. In Proceedings of the 2015 USENIX Annual Technical Conference.

Digital Library

[16]

V. Saraswat, G. Almasi, G. Bikshandi, C. Cascaval, D. Cunningham, D. Grove, S. Kodali, I. Peshansky, and O. Tardieu. 2010. The Asynchronous Partitioned Global Address Space Model. In AMP 10: Proceedings of The First Workshop on Advances in Message Passing.

[17]

P. Sewell, S. Sarkar, S. Owens, F.Z. Nardelli, and M.O. Myreen. 2010. x86-TSO: A Rigorous and Usable Programmer's Model for x86 Multiprocessors. Commun. ACM (2010).

Digital Library

[18]

O Tardieu, B Herta, D Cunningham, D Grove, P Kambadur, V Saraswat, A Shinnar, M Takeuchi, and M Vaziri. 2014. X10 and APGAS at Petascale. In Proceedings of the 19th ACM SIGPLAN symposium on Principles and practice of parallel programming.

Digital Library

[19]

M De Wael, S Marr, B De Fraine, T Van Cutsem, and W De Meuter. 2015. Partitioned Global Address Space Languages. Comput. Surveys (2015).

Digital Library

[20]

DW Walker and JJ Dongarra. 1996. MPI: a standard message passing interface. In Supercomputer.

[21]

M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. 2012. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on networked systems design and implementation.

Digital Library

Cited By

Zhang WCui XFinkler UKingsbury BSaon GKung DPicheny M(2019)Distributed Deep Learning Strategies for Automatic Speech RecognitionICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2019.8682888(5706-5710)Online publication date: May-2019
https://doi.org/10.1109/ICASSP.2019.8682888

Index Terms

DYCE: A Resilient Shared Memory Paradigm for Heterogenous Distributed Systems without Memory Coherence
1. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel algorithms
      1. Massively parallel algorithms

Recommendations

Software Distributed Shared Memory with Transactional Coherence - A Software Engine to Run Transactional Shared-memory Parallel Applications on Clusters
PDP '10: Proceedings of the 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing

Transactional Memory is a novel, promising approach for simplifying parallel programming and increasing its acceptance and diffusion. Until now, almost all the research work on TM has been focused on shared-memory architectures, while very limited ...
Adsmith: An Efficient Object-Based Distributed Shared Memory System On Pvm
ISPAN '96: Proceedings of the 1996 International Symposium on Parallel Architectures, Algorithms and Networks

In this paper, we describe an object-based distributed shared memory called Adsmith. In an object-based DSM, the shared memory consists of many shared objects, through which the shared memory is accessed. Adsmith is built on top of PVM at the library ...
Implementing Distributed Shared Memory on Top of MPI: The DSMPI Library
PDP '96: Proceedings of the 4th Euromicro Workshop on Parallel and Distributed Processing (PDP '96)

Abstract: Distributed shared memory has been recognized as an alternative programming model to exploit the parallelism in distributed memory systems since it provides a higher level of abstraction than simple message passing. DSM combines the simple ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CF'17: Proceedings of the Computing Frontiers Conference

May 2017

450 pages

ISBN:9781450344876

DOI:10.1145/3075564

General Chair:
Roberto Giorgi
University of Siena, IT
,
Program Chairs:
Michela Becchi
North Carolina State University
,
Francesca Palumbo
University of Sassari, IT

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMICRO: ACM Special Interest Group on Microarchitectural Research and Processing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CF '17

Sponsor:

SIGMICRO

CF '17: Computing Frontiers Conference

May 15 - 17, 2017

Siena, Italy

Acceptance Rates

CF'17 Paper Acceptance Rate 43 of 87 submissions, 49%;

Overall Acceptance Rate 273 of 785 submissions, 35%

Upcoming Conference

CF '25

Sponsor:
sigmicro

22nd ACM International Conference on Computing Frontiers

May 28 - 30, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
122
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang WCui XFinkler UKingsbury BSaon GKung DPicheny M(2019)Distributed Deep Learning Strategies for Automatic Speech RecognitionICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2019.8682888(5706-5710)Online publication date: May-2019
https://doi.org/10.1109/ICASSP.2019.8682888

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten