Hello Filippos,
it would sound like it is feasible (note that there also is a PatchMatch wikipedia page 27). A month ago, Francisco Massa suggested a possible approach for implementation that should be relatively easy to follow:

  
    
    
    Need urgent help to implement Patch Match in Pytorch 
  
  
    As I mentioned on Twitter, I’d start with something based on numpy and adapt/optimize for pytorch. Something like https://github.com/jabooth/patchmatch/blob/master/patchmatch.py for example. 
It will be definitely slower than using hand-made kernels, but will definitely be easier to debug. 
Good luck!
  


Of course, we’d love to see your implementation once it is done. 
Best regards
Thomas