I do not think there is a restriction for torch.utils.data.DataLoader. Maybe your network cannot use 5-D data when it expect 4-D?
Personally, when dealing with sets of different rotations, I have been able to “flatten” (using reshape) those into individual images in a batch so my final mini-batch would have the shape (batch_size x N, 3, 64, 64). Does this help?