The error message is most likely pointing to a wrong line, since CUDA operations are called asynchronously.
However, it seems your target values are out of bounds. I assume you are using nn.CrossEntropyLoss, which expects a torch.LongTensor with values in the range [0, nb_classes-1].
Apparently, your smaller dataset does not contain the wrong labels.
You could print your current target tensor and run your code with
CUDA_LAUNCH_BLOCKING=1 python script.py args
to see which target tensor contains the wrong indices.