Your A100s need to use CUDA>=11.0, so the binaries with cudatoolkit==10.1.243 won’t work.
I also see that you’ve MIG enabled on the A100s. Is this on purpose? If so, mask the desired MIG instance via CUDA_VISIBLE_DEVICES and is it in your script as multiple MIG instances cannot be used together.