Hi @ptrblck -  My cost functions are standard MSE loss. I’m not calculating 2nd derivative. This is what I’m doing.

From the forward pass capture the output of the bottleneck layer and the output layer.

Forward pass
output_en,output_de = self.forward(trInput)


Then calculate the reconstruction loss using MSE from the activation from the output layer.
#calculate reconstruction loss
loss = criterion(output_de,trOutput)


Now calculate the cost from the output of the bottleneck layer.
secondCost = self.secondCost(output_en,L)


Now add the second cost with the loss.
loss = loss + secondCost


Now call the backward pass to update parameters.


Backward and optimize
optimizer.zero_grad()
loss.backward()
optimizer.step()
Do you think this is the correct way? I’m unsure whether the parameters of the encoder will be updated or not by the second cost.