Good catch! The difference must be constant (to be defined after de-meaning aka projecting onto the tangent space). Now, if PyTorch doesn’t do this by default, I wonder if training would be better if it did. Some of those reports on the forum… You can do this via register_hook on the input, I’ll certainly try that when I have the time.
Best regards
Thomas