Ah interesting, are you thinking that doing this in-place keeps each minibatch on the GPU (because the DataLoader is passing original references from the source Dataset)?
If this is the case, why doesn’t this happen for a simple image classification dataloader (made from torchvision.datasets, for example):
images = images.cuda()
labels = labels.cuda()

I also forgot to mention that we have to use a custom collate_fn, to get the DataLoader to output batches of a list of image tensors, and a list of dictionaries.
Maybe the default collate_fn argument makes copies of the Dataset examples via torch.stack so it doesn’t alter the device status of the original data in the Dataset.
Will definitely look into this suggestion tonight! Thanks for the tip