Streaming Technologies and Serialization Protocols: Empirical Performance Analysis
<|reference_start|>Streaming Technologies and Serialization Protocols: Empirical Performance Analysis: Efficiently streaming high-volume data is essential for real-time data analytics, visualization, and AI and machine learning model training. Various streaming technologies and serialization protocols have been developed to meet different streaming needs. Together, they perform differently across various tasks and datasets. Therefore, when developing a streaming system, it can be challenging to make an informed decision on the suitable combination, as we encountered when implementing streaming for the UKAEA's MAST data or SKA's radio astronomy data. This study addresses this gap by proposing an empirical study of widely used data streaming technologies and serialization protocols. We introduce an extensible and open-source software framework to benchmark their efficiency across various performance metrics. Our findings reveal significant performance differences and trade-offs between these technologies. These insights can help in choosing suitable streaming and serialization solutions for contemporary data challenges. We aim to provide the scientific community and industry professionals with the knowledge to optimize data streaming for better data utilization and real-time analysis.<|reference_end|>
@article{jackson2024streaming,
    title={Streaming Technologies and Serialization Protocols: Empirical
  Performance Analysis},
    author={Samuel Jackson, Nathan Cummings, Saiful Khan},
    journal={arXiv preprint arXiv:2407.13494},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13494},
    primaryClass={cs.SE cs.NI}
}