Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
Paper
•
2503.09799
•
Published
•
14
Arthur Douillard
ArthurDouillard
AI & ML interests
Continual Learning, Computer Vision, Transformers
Recent Activity
upvoted
a
paper
about 1 month ago
Communication-Efficient Language Model Training Scales Reliably and
Robustly: Scaling Laws for DiLoCo
commented on
a paper
about 1 month ago
Communication-Efficient Language Model Training Scales Reliably and
Robustly: Scaling Laws for DiLoCo
commented on
a paper
about 2 months ago
Streaming DiLoCo with overlapping communication: Towards a Distributed
Free Lunch
Organizations
None yet
ArthurDouillard's activity
upvoted
a
paper
about 1 month ago
commented
a
paper
about 1 month ago
commented
a
paper
about 2 months ago
authored
a
paper
2 months ago
upvoted
a
paper
2 months ago
commented
a
paper
2 months ago
commented
a
paper
3 months ago
upvoted
a
paper
3 months ago
commented
a
paper
3 months ago
upvoted
2
papers
9 months ago
upvoted
a
paper
10 months ago
authored
4
papers
10 months ago
DiLoCo: Distributed Low-Communication Training of Language Models
Paper
•
2311.08105
•
Published
•
15
NEVIS'22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research
Paper
•
2211.11747
•
Published
PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning
Paper
•
2004.13513
•
Published
WARP: On the Benefits of Weight Averaged Rewarded Policies
Paper
•
2406.16768
•
Published
•
23
upvoted
a
paper
10 months ago
authored
a
paper
about 1 year ago
upvoted
a
paper
over 1 year ago
authored
a
paper
over 1 year ago