research-article

Evaluation of NTP/PTP fine-grain synchronization performance in HPC clusters

Authors:

Andrea Bartolini,

Daniele Cesarini,

Luca BeniniAuthors Info & Claims

ANDARE '18: Proceedings of the 2nd Workshop on AutotuniNg and aDaptivity AppRoaches for Energy efficient HPC Systems

Article No.: 3, Pages 1 - 6

https://doi.org/10.1145/3295816.3295819

Published: 04 November 2018 Publication History

Abstract

Fine-grain time synchronization is important to address several challenges in today and future High Performance Computing (HPC) centers. Among the many, (i) co-scheduling techniques in parallel applications with sensitive bulk synchronous workloads, (ii) performance analysis tools and (iii) autotuning strategies that want to exploit State-of-the-Art (SoA) high resolution monitoring systems, are three examples where synchronization of few microseconds is required. Previous works report custom solutions to reach this performance without incurring in extra cost of dedicated hardware. On the other hand, the benefits to use robust standards which are widely supported by the community, such as Network Time Protocol (NTP) and Precision Time Protocol (PTP), are evident. With today's software and hardware improvements of these two protocols and off-the-shelf integration in SoA HPC servers no expensive extra hardware is required anymore, but an evaluation of their performance in supercomputing clusters is needed. Our results show NTP can reach on computing nodes an accuracy of 2.6 μs and a precision below 2.7 μs, with negligible overhead. These values can be bounded below microseconds, with PTP and low-cost switches (no needs of GPS antenna). Both protocols are also suitable for data time-stamping in SoA HPC monitoring infrastructures. We validate their performance with two real use-cases, and quantify scalability and CPU overhead. Finally, we report software settings and low-cost network configuration to reach these high precision synchronization results.

References

[1]

2008. IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems.

[2]

W. Abu Ahmad et al. 2017. Design of an Energy Aware Petaflops Class High Performance Cluster Based on Power Architecture. In 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). 964--973.

[3]

A. Bartolini et al. 2018. The D.A.V.I.D.E. Big-Data-Powered Fine-Grain Power and Performance Monitoring Support. In ACM International Conference on Computing Frontiers 2018.

Digital Library

[4]

D. Becker et al. 2008. Implications of non-constant clock drifts for the timestamps of concurrent events. In 2008 IEEE International Conference on Cluster Computing. 59--68.

[5]

D. Becker et al. 2009. Scalable timestamp synchronization for event traces of message-passing applications. Parallel Comput. 35, 12 (2009), 595 -- 607. Selected papers from the 14th European PVM/MPI Users Group Meeting.

Digital Library

[6]

D. Becker et al. 2010. Synchronizing the Timestamps of Concurrent Events in Traces of Hybrid MPI/OpenMP Applications. In 2010 IEEE International Conference on Cluster Computing.

Digital Library

[7]

J. Burbank et al. 2015. Network Time Protocol Version 4: Protocol and Algorithms Specification. IETF RFC 5905.

[8]

CISCO Systems, Inc. 2016. Cisco Industrial Ethernet 3000 Layer 2/Layer 3 Series Switches. Datasheet.

[9]

C. Conficoni et al. 2015. Energy-aware cooling for hot-water cooled supercomputers. In 2015 Design, Automation Test in Europe Conference Exhibition (DATE). 1353--1358.

Digital Library

[10]

B. Ferencz et al. 2013. Hardware assisted COTS IEEE 1588 solution for x86 Linux and its performance evaluation. In 2013 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication (ISPCS) Proceedings.

[11]

P. Giannozzi et al. 2009. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics: Condensed Matter 21, 39 (2009), 395502.

[12]

T. Ilsche et al. 2018. Power measurement techniques for energy-efficient computing: reconciling scalability, resolution, and accuracy. Computer Science - Research and Development (Apr 2018).

[13]

T. Jones et al. 2017. An evaluation of the state of time synchronization on leadership class supercomputers. Concurrency and Computation: Practice and Experience 30, 4 (2017), e4341. e4341 cpe.4341.

[14]

T. Jones and G. A. Koenig. 2012. Clock synchronization in high-end computing environments: a strategy for minimizing clock variance at runtime. Concurrency and Computation: Practice and Experience 25, 6 (2012), 881--897.

[15]

A. Libri et al. 2016. Evaluation of synchronization protocols for fine-grain HPC sensor data time-stamping and collection. In 2016 International Conference on High Performance Computing Simulation (HPCS). 818--825.

[16]

A. Libri et al. 2018. Dwarf in a Giant: Enabling Scalable, High-Resolution HPC Energy Monitoring for Real-Time Profiling and Analytics. ArXiv e-prints (June 2018). arXiv:cs.DC/1806.02698

[17]

Mellanox Technologies. 2015. EDR InfiniBand. OFA UM 2015, OpenFabrics Software User Group Workshop.

[18]

J. Ridoux and D. Veitch. 2009. Ten Microseconds Over LAN, for Free (Extended). IEEE Transactions on Instrumentation and Measurement 58, 6 (June 2009), 1841--1848.

[19]

J. Serrano et al. 2013. THE WHITE RABBIT PROJECT. In Proceedings of IBIC 2013, Oxford, UK. http://cds.cern.ch/record/1743073

[20]

C. Silvano et al. 2017. The ANTAREX Tool Flow for Monitoring and Autotuning Energy Efficient HPC Systems. In SAMOS 2017 - International Conference on Embedded Computer Systems: Architecture, Modeling and Simulation. Pythagorion, Greece. https://hal.inria.fr/hal-01615945

Cited By

Schuchart JHunold SBosilca G(2023)Synchronizing MPI Processes in Space and TimeProceedings of the 30th European MPI Users' Group Meeting10.1145/3615318.3615325(1-11)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3615318.3615325
Megías CVázquez VRos ECappelli MDíaz J(2023)Ethernet-based timing system for accelerator facilitiesComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109897233:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109897
Schmetz ALee TZontar DBrecher C(2022)The Time Synchronization Problem in data-intense ManufacturingProcedia CIRP10.1016/j.procir.2022.05.070107(827-832)Online publication date: 2022
https://doi.org/10.1016/j.procir.2022.05.070
Show More Cited By

Evaluation of NTP/PTP fine-grain synchronization performance in HPC clusters

Recommendations

A Time Synchronization Method for NTP
RTCSA '99: Proceedings of the Sixth International Conference on Real-Time Computing Systems and Applications

This paper introduces an implementation of a clock synchronization method for NTP(Network Time Protocol). NTP is widely used and an effective application protocol in maintaining time synchronization over the network. But the performance of NTP depends ...
Research on High Precision Time Synchronization of Android Based on NTP
ICIIP '19: Proceedings of the 4th International Conference on Intelligent Information Processing

Network Time Protocol (NTP) is one of the key technologies for time synchronization of mobile devices. It is very necessary for its research on Android platform. This paper designs an Android time synchronization system based on NTP for the requirement ...
On the Evaluation of Clock Synchronization Methods for Networked Control Systems
ECBS-EERC '13: Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based Systems

A distribute control requires to maintain time synchronization to the degree of precision depending on a particular application domain. Many large distributed systems employ Ethernet or even full TCP/IP stack for communication. Two prevalent protocols ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ANDARE '18: Proceedings of the 2nd Workshop on AutotuniNg and aDaptivity AppRoaches for Energy efficient HPC Systems

November 2018

36 pages

ISBN:9781450365918

DOI:10.1145/3295816

General Chairs:
Andrea Bartolini
University of Bologna, Italy
,
João M. P. Cardoso
Faculdade de Engenharia da Universidade do Porto, Portugal
,
Cristina Silvano
Politecnico di Milano, Italy

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

EC
E4 Computer Engineering SpA

Conference

ANDARE'18

ANDARE'18: 2nd Workshop on AutotuniNg and aDaptivity AppRoaches for Energy efficient HPC Systems

November 4, 2018

Limassol, Cyprus

Acceptance Rates

Overall Acceptance Rate 3 of 4 submissions, 75%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
209
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)3

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Schuchart JHunold SBosilca G(2023)Synchronizing MPI Processes in Space and TimeProceedings of the 30th European MPI Users' Group Meeting10.1145/3615318.3615325(1-11)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3615318.3615325
Megías CVázquez VRos ECappelli MDíaz J(2023)Ethernet-based timing system for accelerator facilitiesComputer Networks: The International Journal of Computer and Telecommunications Networking10.1016/j.comnet.2023.109897233:COnline publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1016/j.comnet.2023.109897
Schmetz ALee TZontar DBrecher C(2022)The Time Synchronization Problem in data-intense ManufacturingProcedia CIRP10.1016/j.procir.2022.05.070107(827-832)Online publication date: 2022
https://doi.org/10.1016/j.procir.2022.05.070
Kim HIshikawa M(2021)Sub-Frame Evaluation of Frame Synchronization for Camera Network Using Linearly Oscillating Light SpotSensors10.3390/s2118614821:18(6148)Online publication date: 13-Sep-2021
https://doi.org/10.3390/s21186148
Libri ABartolini ABenini L(2021)DiG: enabling out-of-band scalable high-resolution monitoring for data-center analytics, automation and control (extended)Cluster Computing10.1007/s10586-020-03219-7Online publication date: 7-Jan-2021
https://doi.org/10.1007/s10586-020-03219-7
Torres LBarrios C(2021)Methodology for Design and Implementation an Efficient HPC ClusterHigh Performance Computing10.1007/978-3-030-68035-0_6(71-85)Online publication date: 3-Feb-2021
https://doi.org/10.1007/978-3-030-68035-0_6
Libri ABartolini ABenini L(2020) pAElla : Edge AI-Based Real-Time Malware Detection in Data Centers IEEE Internet of Things Journal10.1109/JIOT.2020.29867027:10(9589-9599)Online publication date: Oct-2020
https://doi.org/10.1109/JIOT.2020.2986702
Borghesi ALibri ABenini LBartolini A(2019)Online Anomaly Detection in HPC Systems2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)10.1109/AICAS.2019.8771527(229-233)Online publication date: Mar-2019
https://doi.org/10.1109/AICAS.2019.8771527

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents