Did you actually build pytorch from source? or just used the prebuilt  libtorch? It should be as simple as having that amd-blas library within your main directory for libs, and running the build process. Pytorch will find the BLAS package as long as it is there in the main usr libs.
Follow the build process here to build from source

  
      
      GitHub
  
  
    

pytorch/pytorch 20

Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch