@11116 I see you’re using Apex. The error message in the epilogue means that not all learnable parameters in your model had their gradients computed (i.e. they didn’t participate in the forward pass). This is possible if you use any type of control flow in your forward pass that excludes use of certain parameters. You can fix this in Apex by delaying all reduction until the very end of the backwards pass by using the delay_allreduce option. See https://nvidia.github.io/apex/parallel.html 16 for more details.