Breaking the Memory Barrier: Near Infinite Batch Size Scaling for
  Contrastive Loss
Contrastive loss is a powerful approach for representation learning, where
larger batch sizes enhance performance by providing more negative samples to
better distinguish between similar and dissimilar data. However, scaling batch
sizes is constrained by the quadratic growth in GPU memory consumption,
primarily due to the full instantiation of the similarity matrix. To address
this, we propose a tile-based computation strategy that partitions the
contrastive loss calculation into arbitrary small blocks, avoiding full
materialization of the similarity matrix. Furthermore, we introduce a
multi-level tiling strategy to leverage the hierarchical structure of
distributed systems, employing ring-based communication at the GPU level to
optimize synchronization and fused kernels at the CUDA core level to reduce I/O
overhead. Experimental results show that the proposed method scales batch sizes
to unprecedented levels. For instance, it enables contrastive training of a
CLIP-ViT-L/14 model with a batch size of 4M or 12M using 8 or 32 A800 80GB
without sacrificing any accuracy. Compared to SOTA memory-efficient solutions,
it achieves a two-order-of-magnitude reduction in memory while maintaining
comparable speed. The code will be made publicly available.
671b005026d87414f5a61579
Zesen Cheng
claimed_verified
2024-10-25T09:30:02.067Z
65b2529285b6c21448a10d65
/avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
Zesen Cheng
user
ClownRat
671b005026d87414f5a6157a
Hang Zhang
671b005026d87414f5a6157b
Kehan Li
671b005026d87414f5a6157c
Sicong Leng
671b005026d87414f5a6157d
Zhiqiang Hu
claimed_verified
2024-10-25T09:30:00.203Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
user
Zhiqiang007
671b005026d87414f5a6157e
Fei Wu
671b005026d87414f5a6157f
Deli Zhao
671b005026d87414f5a61580
Xin Li
671b005026d87414f5a61581
Lidong Bing
claimed_verified
2024-10-25T09:29:58.206Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
Breaking the Memory Barrier: Near Infinite Batch Size Scaling for
  Contrastive Loss
Contrastive loss is a powerful approach for representation learning, where
larger batch sizes enhance performance by providing more negative samples to
better distinguish between similar and dissimilar data. However, scaling batch
sizes is constrained by the quadratic growth in GPU memory consumption,
primarily due to the full instantiation of the similarity matrix. To address
this, we propose a tile-based computation strategy that partitions the
contrastive loss calculation into arbitrary small blocks, avoiding full
materialization of the similarity matrix. Furthermore, we introduce a
multi-level tiling strategy to leverage the hierarchical structure of
distributed systems, employing ring-based communication at the GPU level to
optimize synchronization and fused kernels at the CUDA core level to reduce I/O
overhead. Experimental results show that the proposed method scales batch sizes
to unprecedented levels. For instance, it enables contrastive training of a
CLIP-ViT-L/14 model with a batch size of 4M or 12M using 8 or 32 A800 80GB
without sacrificing any accuracy. Compared to SOTA memory-efficient solutions,
it achieves a two-order-of-magnitude reduction in memory while maintaining
comparable speed. The code will be made publicly available.
A tile-based computation strategy optimizes contrastive loss by partitioning calculations, enabling large batch sizes without significant memory or speed sacrifices.
contrastive loss representation learning similarity matrix tile-based computation multi-level tiling ring-based communication fused kernels CLIP-ViT-L/14 GPU memory consumption