If I set batch-size to 256 and use all of the GPUs on my system (lets say I have 8), will each GPU get a batch of 256 or will it get 256//8 ?
If my memory serves me correctly, in Caffe, all GPUs would get the same batch-size , i.e 256 and the effective batch-size would be 8*256 , 8 being the number of GPUs and 256 being the batch-size.
Is the outcome/answer any different when using .cuda() (with no parameters) and when using torch.nn.DataParallel(model, device_ids=args.gpus) (specifying which gpu ids to use)?
Thanks a lot in advance