This error indicates that your device is running out of memory and crashes.
This could happen, e.g. in case you are storing unnecessary tensors and are thus increasing the memory usage throughout the training (check the memory usage during the training via nvidia-smi and make sure it’s constant after some iterations/epochs), or e.g. if you are working with variable input shapes and one particular batch contains especially large inputs (use a max. shape in that case), or e.g. if you are using a shared system and another user uses the same GPU and allocated memory, etc.