You could use the Android build as inspiration.
That said, you might also be able to use @LeviViana 's  build for the Raspberry and take the libtorch included in PyTorch:

  
    
    
    PyTorch 1.3 wheels for Raspberry Pi (Python 3.7) quantization
  
  
    I’ve made it available here: https://wintics-opensource.s3.eu-west-3.amazonaws.com/torch-1.3.0a0%2Bdeadc27-cp37-cp37m-linux_armv7l.whl. 
Have fun !
  


Best regards
Thomas