Solved by ptrblck in post #4 
              
              
                Both approaches should yield the same result and I assume your models might have used different layer parameters. 
This code snippet shows that the different forward implementations create the same gradients up the floating point precision: 
import torch
torch.manual_seed(0)
import torch.nn as nn
im…