I think I fixed my own problem, but posting the answer here in case anybody ever comes across this later down the line.
The first issue comes from the fact that (at least in my case) the A100 GPUs comes with sm_80, but also with pre-installed CUDA 11+. In this case, I was sabotaging myself by module loading CUDA 10.1 and cudnn 7.6.5.32 on the cluster.
The solution came by using the CUDA 11.1 version of pytorch from here: Start Locally | PyTorch 198
pip3 install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
Then, I just stopped loading CUDA 10.1 and cudnn 7.6.5 as separate modules, which made all errors go away. Running a few tests jobs now, and will post any updates in case other things get broken.