I guess your GPU is running out of memory during the model training, not while creating the batch.
If so, you could split the batch into chunks in the DataLoader loop and perform multiple training steps.
Alternatively, you could only load the desired number of samples in the Dataset and create a batch with the smaller number of samples. Removing samples in the collate_fn could work, but would be wasteful as previously described.