this seems like a cudnn bug.
Can you give me a way to reproduce this on my side?
You can also try setting this flag (at the beginning of your script) and see if it goes away: torch.backends.cudnn.benchmark=True