Thank you for your response. Sorry for my naive thoughts. I am still quite don’t understand well.
For example, if my ground truth is in [0, 1]. After the final fully connected layer, I would like to scale the output into [0, 1]. So I add a Sigmoid function after FC layer. And I will compare the prediction with the ground truth and based on the loss to do the backpropagation to update the network.
If during the training, I only use the final FC layer to output the prediction and the network fits well. However, during the testing stage, I add a Sigmoid to squash the FC output. Maybe in the end the original performance is good but with the Sigmoid function, the performance might not be as good as the original backpropagation one?
Thank you again for your response. Maybe my idea and understanding are naive and trivial.