In that case you want to make sure you have the latest cuda/cudnn installed.
For distributed, the gloo backend should be already properly provided.
For mixed precision I think basic cuda libs handle that well.
For linear algebra. You want a good blas/lapack lib on cpu like mkl or openblas. And for gpu you will need magma.
You can find some instructions here: https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md 28