I don’t think you will see a significant speedup, if you use libtorch, but please update this thread in case you decide to try it out.
Since you are mostly using PyTorch methods in your code, there shouldn’t be a lot of Python overhead.