Hello,
I hope this is the right forum to post my issue.
During the model training I received the following error message with an additional code snippet to reconstruct the error. The code snippet throws the same error.
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([1, 256, 128, 128], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(256, 256, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

If I set

torch.backends.cudnn.benchmark = False

the error is not triggered. Originally, the error was triggered when I used transforms.RandomCrop(256) for the training data and transforms.RandomCrop(512) for the validation data. With the same crop size the error is not triggered.
I don’t know if this is a bug or if I did something wrong.
BR,
Patrick