Solved by ptrblck in post #2 
              
              
                Thanks for the great code and the interesting question! 
Based on your output (and my runs) it seems that the bias of the preceding linear layer is not updated. 
While the check claims so, you could check the gradients of the bias parameter and would see that these gradients are indeed really small: …