Thanks for the model. I just verified that it is failing and mkldnn does change the model layers. I don’t have a lot of context on mkl dnn, but I created an issue on github to track this and loop in the right people, Support for mkldnn + ddp · Issue #56024 · pytorch/pytorch · GitHub.