You could try to use a static input (e.g. torch.ones) and compare the outputs between the Python frontend and libtorch. If these values differ by a larger error it would mean that the model itself is not performing the same operations and you should look into the forward definition to isolate the difference.
On the other hand, if the outputs are equal for a static input, the issue would most likely be in the data loading and preprocessing.