RylanSchaeffer:

I’m not sure I understand how cross entropy relates to geometric and arithmetic means.


In wikipedia’s page linked above, see “relation to log-likelihood” section.
Then consider a dataset with two samples, you optimize their likelihoods independently, and total likelihood is represented by their product, P_total = P1*P2, or log(P_total) = log(P1)+log(P2). So, geometric mean of P1 and P2, or log-average of likelihoods is what “sample loss” normally implicitly maximizes. In other words, increasing smaller likelihoods has bigger effect, due to log transform.
Now if you optimize P1+P2, it is possible to get to the same parameters, but optimization path will be different, less robust I think.