Your model and training might use too much device memory, which will yield this error.
Try to reduce the batch size to lower the memory footprint.
If that’s not possible, you could try to use torch.utils.checkpoint 8 to trade compute for memory.
Also, check the available memory of your GPU via nvidia-smi and, if possible, stop other processes which might take some memory.