Towards cloud-native scientific workflow management
<|reference_start|>Towards cloud-native scientific workflow management: Cloud-native is an approach to building and running scalable applications in modern cloud infrastructures, with the Kubernetes container orchestration platform being often considered as a fundamental cloud-native building block. In this paper, we evaluate alternative execution models for scientific workflows in Kubernetes. We compare the simplest job-based model, its variant with task clustering, and finally we propose a cloud-native model based on microservices comprising auto-scalable worker-pools. We implement the proposed models in the HyperFlow workflow management system, and evaluate them using a large Montage workflow on a Kubernetes cluster. The results indicate that the proposed cloud-native worker-pools execution model achieves best performance in terms of average cluster utilization, resulting in a nearly 20\% improvement of the workflow makespan compared to the best-performing job-based model. However, better performance comes at the cost of significantly higher complexity of the implementation and maintenance. We believe that our experiments provide a valuable insight into the performance, advantages and disadvantages of alternative cloud-native execution models for scientific workflows.<|reference_end|>
@article{orzechowski2024towards,
    title={Towards cloud-native scientific workflow management},
    author={Michal Orzechowski, Bartosz Balis, Krzysztof Janecki},
    journal={arXiv preprint arXiv:2408.15445},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.15445},
    primaryClass={cs.DC}
}