hi, Xiao Liang. you need to install MKLDNN from conda first and then follow the installation steps from README.
conda install -c mingfeima mkldnn
Performance speedup you can get strongly depends on your CPU type. The optimization from MKLDNN is mostly for Xeon CPU. We also have some BKM 40 for better CPU performance. i upload some benchmark numbers at link 48. you can take a look.
The optimization job is still working in progress and we will continuously boost up cpu performance. Conv performance should be much faster with MKLDNN, rnn optimization is currently under review.
If you run on Xeon CPU, NNPACK is definitely not as performant as MKLDNN, we double checked this before.
Let me know if you have trouble compiling with MKLDNN.