Solved by ptrblck in post #4 
              
              
                Thanks for the code. 
The difference is indeed in the usage of the batch norm layers. 
While you are initializing two different nn.BatchNorm1d layers in your sequential approach, you are reusing the same in your manual approach. 
Could you create two separate batch norm layers and run the test again…