I fixed the GPU memory by installing the cudnn in anaconda. Compiling the pytorch from source doesn’t install cudnn either, which needs us to further install it by our own.
However, the high CPU performance issue still exists.