My neural network has only one output, instead of using Non-linear activations in hidden layers, the output of the neural network is the production of the neurons from the last layer. When the weights are initialized by normal distribution (mean is 0 and std is 1), if I increase the neuron numbers of the last layer, the production (output) can be very large. Such large numbers can cause double precision out of range, this is what I named overflowing.
So, I want to find a way to standardizing the neuron outputs of the last layer, which may not cause overflowing.