OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
<|reference_start|>OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training: OpenDiLoCo is an open-source implementation and replication of the Distributed Low-Communication (DiLoCo) training method for large language models. We provide a reproducible implementation of the DiLoCo experiments, offering it within a scalable, decentralized training framework using the Hivemind library. We demonstrate its effectiveness by training a model across two continents and three countries, while maintaining 90-95% compute utilization. Additionally, we conduct ablations studies focusing on the algorithm's compute efficiency, scalability in the number of workers and show that its gradients can be all-reduced using FP16 without any performance degradation. Furthermore, we scale OpenDiLoCo to 3x the size of the original work, demonstrating its effectiveness for billion parameter models.<|reference_end|>
@article{jaghouar2024opendiloco:,
    title={OpenDiLoCo: An Open-Source Framework for Globally Distributed
  Low-Communication Training},
    author={Sami Jaghouar, Jack Min Ong and Johannes Hagemann},
    journal={arXiv preprint arXiv:2407.07852},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07852},
    primaryClass={cs.LG cs.DC}
}