Solved by ptrblck in post #4 
              
              
                Yes, that’s correct. Additionally to this the intermediate activations were also calculated using the “old” parameters. The backward pass would use the new parameters with the old intermediates to calculate the gradients for the new parameters, which would be wrong. 

Yes, that would be an alternat…