Solved by ptrblck in post #10 
              
              
                Thanks for the code. 
Using your code and just passing some random values into the models, I can get valid gradients in both models: 
model = Network(2, 2)
modelFCNN = NetworkFCNN(2, 1)
criterion1 = torch.nn.MSELoss()
    
data = torch.randn(1, 2)
y_pred_ref = modelFCNN(data)

# concatenate the tens…