libcudnn.so and cudnn.h have to be in their respective CUDNN_LIB_DIR and CUDNN_INCLUDE_DIR, make sure that this is the case.
Cudnn detection during compilation is done in this file https://github.com/pytorch/pytorch/blob/master/tools/setup_helpers/cudnn.py 82, make sure that cudnn is actually found when you compile.