Yes exactly, that is why in Dataset definition, there is a function called __getitem__ which only returns a single item of dataset after augmentation. Then dataloaders take advantage of this and can run this method in parallel. And after this, you convert tensors in batch, obtained from dataloader to tensors in GPU so this pipeline will be held.