Accelerating Time-to-Science by Streaming Detector Data Directly into Perlmutter Compute Nodes
<|reference_start|>Accelerating Time-to-Science by Streaming Detector Data Directly into Perlmutter Compute Nodes: Recent advancements in detector technology have significantly increased the size and complexity of experimental data, and high-performance computing (HPC) provides a path towards more efficient and timely data processing. However, movement of large data sets from acquisition systems to HPC centers introduces bottlenecks owing to storage I/O at both ends. This manuscript introduces a streaming workflow designed for an high data rate electron detector that streams data directly to compute node memory at the National Energy Research Scientific Computing Center (NERSC), thereby avoiding storage I/O. The new workflow deploys ZeroMQ-based services for data production, aggregation, and distribution for on-the-fly processing, all coordinated through a distributed key-value store. The system is integrated with the detector's science gateway and utilizes the NERSC Superfacility API to initiate streaming jobs through a web-based frontend. Our approach achieves up to a 14-fold increase in data throughput and enhances predictability and reliability compared to a I/O-heavy file-based transfer workflow. Our work highlights the transformative potential of streaming workflows to expedite data analysis for time-sensitive experiments.<|reference_end|>
@article{welborn2024accelerating,
    title={Accelerating Time-to-Science by Streaming Detector Data Directly into
  Perlmutter Compute Nodes},
    author={Samuel S. Welborn, Bjoern Enders, Chris Harris, Peter Ercius, Deborah
  J. Bard},
    journal={arXiv preprint arXiv:2403.14352},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14352},
    primaryClass={cs.NI cs.DC}
}