Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3624062.3624122acmotherconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

Report on Adaptable Open-Source Disaster Recovery Solution for Multi-Petabyte Storage Systems

Published: 12 November 2023 Publication History

Abstract

The current generation of Research Data Store (RDS) at The University of Sydney comprises a pair of peta-scale data storage systems. We implemented a disaster recovery (DR) solution on both systems for data protection against catastrophic failure. To handle large amount of data transactions into RDS, we took an open-source approach to design an adaptable DR solution that enables parallelized data replication capability between the pair of storage systems. In the last three years of operations, the DR solution has gone through a few iterations which saw improvement in efficiency. In this paper, we present the findings and outcomes from our DR implementation.

Supplemental Material

MP4 File
Recording of "Report on Adaptable Open-Source Disaster Recovery Solution for Multi-Petabyte Storage Systems" presentation at HPCSYSPROS23.
SH File
Data replication workflow script

References

[1]
H. Leong, A. Janke, S. Kolmann and D. Richards, “Parallelized Data Replication of Multi-Petabyte Storage Systems,” in SC (SC20) ACM SIGHPC SYSPROS Workshop 2020, Atlanta, 2020. https://doi.org/10.5281/zenodo.4737878
[2]
A. Tridgell and W. Davison, “The rsync algorithm,” The Australian National University, Canberra, 1996.
[3]
O. Tange, GNU Parallel 2018, Ole Tange, 2018. https://doi.org/10.5281/zenodo.1146014
[4]
D. Sikich, G. D. Natale, M. LeGendre and A. Moody, “mpiFileUtils: A Parallel and Distributed Toolset for Managing Large Datasets,” in 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems, Denver, 2017.

Index Terms

  1. Report on Adaptable Open-Source Disaster Recovery Solution for Multi-Petabyte Storage Systems

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SC-W '23: Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis
    November 2023
    2180 pages
    ISBN:9798400707858
    DOI:10.1145/3624062
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 November 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Disaster recovery
    2. data replication
    3. data transfer
    4. parallel

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SC-W 2023

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 18
      Total Downloads
    • Downloads (Last 12 months)9
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media