Automated, Reliable, and Efficient Continental-Scale Replication of 73 Petabytes of Climate Simulation Data: A Case Study
<|reference_start|>Automated, Reliable, and Efficient Continental-Scale Replication of 73 Petabytes of Climate Simulation Data: A Case Study: We report on our experiences replicating 7.3 petabytes (PB) of Earth System Grid Federation (ESGF) climate simulation data from Lawrence Livermore National Laboratory (LLNL) in California to Argonne National Laboratory (ANL) in Illinois and Oak Ridge National Laboratory (ORNL) in Tennessee. This movement of some 29 million files, twice, undertaken in order to establish new ESGF nodes at ANL and ORNL, was performed largely automatically by a simple replication tool, a script that invoked Globus to transfer large bundles of files while tracking progress in a database. Under the covers, Globus organized transfers to make efficient use of the high-speed Energy Sciences network (ESnet) and the data transfer nodes deployed at participating sites, and also addressed security, integrity checking, and recovery from a variety of transient failures. This success demonstrates the considerable benefits that can accrue from the adoption of performant data replication infrastructure.<|reference_end|>
@article{lacinski2024automated,,
    title={Automated, Reliable, and Efficient Continental-Scale Replication of 7.3
  Petabytes of Climate Simulation Data: A Case Study},
    author={Lukasz Lacinski and Lee Liming and Steven Turoscy and Cameron Harr and
  Kyle Chard and Eli Dart and Paul Durack and Sasha Ames and Forrest M. Hoffman
  and Ian T. Foster},
    journal={arXiv preprint arXiv:2404.19717},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.19717},
    primaryClass={cs.DC}
}