Hi Tom,
Thank you for your reply.
I read your answer to this issue 6, regarding non-deterministic behavior during backward(), which is exactly where I located the issue at the moment. All values remain unchanged in forward propagation and the gradient difference appears in both encoder and decoders.
Though I don’t understand how my test machine with a GTX 980T would display 1e-10 or even 1e-12 difference for the gradients, should I trust such minimal difference? Certainly I can’t ignore them as the difference in weights would grow bigger later.
A rookie question - I reproduced example code form pytorch to make sure nothing else caused the problem. But I found the encoder and decoder pair from here 3 caused most trouble after backward().
I applied all available deterministic methods mentioned in the post, by reducing the conv and res layers in the code I mentioned above, I could see less occurrence of divergence but most of the gradient difference are at level of 1e-4 or 1e-3, isn’t that too big too ignore?
But thank you for your feedback and great work.
Best reagards,
CY