ShiweiJin:

add a Sigmoid to the output during both training and testing stages


You can do that after each training epoch/iteration, but I believe that shouldn’t  be part of the training itself. That is, suppose you would like to measure the performance of the model on the training data after a few epochs, then you can add the Sigmoid to scale the output. You should not back-propagate through the Sigmoid, as training will be severely deterministic. Best way is to try both approaches and see what happens, ie to add or not to add the Sigmoid during training.
Update: I am assuming the use of a loss function that embeds an activation function, like the Softmax loss.
What the Sigmoid does is a kind of squashing operation.
sigmiod1200×800 27 KB