Let’s consider the batch size is 64 and we want to run in a cluster of two nodes, where each node contains 4 GPUs. How the batch will be split? Will the batch first split into two and thus each node will get a batch of data size 32, and finally, each node will split the data among the four GPUs, thus each GPU will get a batch of data size 8? is this the way the will be split in DistributedDataParallel mode?
Thanks in advance for the clarification.