Probably the backward of the downsampling:
https://pytorch.org/docs/stable/notes/randomness.html 16
Seems to be one of the things everyone wants to have but nobody takes the time to do.
Best regards
Thomas