I’m interested in trying to reproduce this on 10.2/11.x. Do you have a minimal example handy (e.g., just the model with random data) that demonstrates the issue?
The driver itself probably wouldn’t be responsible for the issue. But the switch from CUDA 10.2 torch to 11.x very well might be the cause.