Could you post some information for this setup, i.e.:

which GPU are you using
how did you install CUDA11 and which version exactly
which cudnn version are you using
which PyTorch commit are you using
are you seeing the extremely high loss after a certain number of iterations for random inputs in [0, 1]?