Actually, it has been experimentally shown that this helps the models to converge faster or achieve better result.
At inference time, you need to use same mean and std you have used for training, then the final outputs depend on your network may need unnormalization with same mean and std to get back to original scale.