If I load the weights normally, then convert my model(net) to mkldnn, and then convert the input to the mkldnn, and then try to feed forward it, it fails with the following message :
(base) user@F3S-Hasanpour:/mnt/d/Codes/Pytorch_Retinaface_MKLDN$ cd /mnt/d/Codes/Pytorch_Retinaface_MKLDN ; env PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /home/user/anaconda3/bin/python /home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py --default --nodebug --client --host localhost --port 50776 /mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py 
Loading pretrained model from ./weights/mobilenet0.25_Final.pth
remove prefix 'module.'
input type for img_path: str
Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 327, in <module>
    model_weights=model.weight )
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 142, in detect
    loc, conf, landms = net(img)  # forward pass
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/retinaface.py", line 225, in forward
    fpn = self.fpn(out)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/models/net.py", line 84, in forward
    output1 = self.output1(input[0])
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (Mkldnntorch.FloatTensor) and weight type (torch.FloatTensor) should be the same


and if I convert the weights into the mkldnn as well I get:
default --nodebug --client --host localhost --port 50734 /mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py 
source /home/user/anaconda3/bin/activate
conda activate base
Loading pretrained model from ./weights/mobilenet0.25_Final.pth
remove prefix 'module.'
Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 327, in <module>
    model_weights=model.weight )
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 96, in detect
    net = load_model(net, trained_model, cpu)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 67, in load_model
    pretrained_dict = to_mkldnn(pretrained_dict)
  File "/mnt/d/Codes/Pytorch_Retinaface_MKLDN/test_detect_align_MKLDNN.py", line 51, in to_mkldnn
    state_dict_new[k] = p.to_mkldnn()
RuntimeError: cpu_tensor.scalar_type() == ScalarType::Float INTERNAL ASSERT FAILED at /pytorch/aten/src/ATen/native/mkldnn/MKLDNNConversions.cpp:28, please report a bug to PyTorch. dense_to_mkldnn expects float tensor input