Strange. Are you using a custom dataset written by you or a dataset from torchvision?
If you are using a custom dataset, can you verify __getitem__ method if you are handling the indices correctly?
Would you be able to provide a toy example to reproduce the issue?