I ran into a strange problem while compiling pytorch from source to support MPI backend.
My setup:
4 rtx 2080ti GPU
pytorch 1.6.0
cuda version 10.1
The problem happens when I tried to compile pytorch 1.6.0 from source.
I followed the steps from the pytorch website. GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration 2
The compilation when smoothly and I was able to see all the 4 GPUs on my machine.
But when I tested with a simple code that I have been using for a while, I got the following error message.
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
The same code runs perfectly when I use the pytorch install directly from pip on the same machine.
Can anyone give some advices to fix this problem?