Thank you @osalpekar. But, if the batch size remains the same across the multiple GPUs, where is the distribution of training happening? I mean which part of the memory load on a single GPU is getting split across multiple GPUs? Please let me know if I am missing something.