Why in some neural network training cases the outputs are assumed to be a probability distribution? This might be a stupid question but this question is bugging me for a long time. When I first started working with neural networks we usually created a neural network which output vector of number values. We then used crossentropy or MSE loss for training classification and regression models respectively.
But now many generative models assumes that the output of neural network as a probability distribution and uses loss function such as Kullback Liebler (KL) Divergence, Jensen-Shannon divergence to train a model, example: GANs, VAEs etc.
Why in some training cases the outputs are assumed to be a probability distribution? What is the advantages and disadvantages of this assumption?

When using cross-entropy for classification tasks, that can also be viewed as a conditional probability distribution (categorical - the output values are the parameters giving the probability of each catageory) and when using MSE for regression that is also specifying a conditional probability distribution (the conditional mean of a Gaussian distribution - but with a fixed variance).  The loss function is a good way of incorporating expert knowledge in the design of the model.  I have found this useful, for example in modelling the conditional variance of the response variable, which is not constant in some applications.
