On my tiny laptop GPU, in a training loop, pushing each batch one at a time onto the GPU with default number of workers (I guess 1) takes 2 or 3 more time than pushing the whole dataset once then manually creating batches in the traning loop. Same thing on google cobab GPU.
I will try to play with mutliples workers and pin_memory on google colab.
Thanks for you help. Take care.