Some resources that you may find useful:


Distributed training: Note about dataset batching - Multi-GPU and distributed training 2


TPUs: Rule of thumb: pick efficient values for batch and feature dimensions - Performance Guide  |  Cloud TPU  |  Google Cloud 1


TPUs: Batch size, learning rate, steps_per_execution - Tensor Processing Units (TPUs) Documentation | Kaggle 1


Multi-GPU training: “batch size considerations depend on your training framework” - Training With Mixed Precision :: NVIDIA Deep Learning Performance Documentation


“Increasing batch size while mitigating accuracy degradation is actively researched in the ML and systems communities” (Augment your batch: better training with larger batches - Hoffer et al., 2019)


A Recipe for Training Neural Networks 1 (2019, by Andrej Karpathy)