Why do we handpick a specific loss for classification I get the MLE log likelihood to get a "good loss", and that DL models have non convex losses, thus leading to have local minima and so on.
However, my point is a bit different. Assume that we have a NN, and we want to do binary classification.
Independently if we use MAE, MSE, BCE, having 0 loss would mean that we are predicting always perfectly the exact label. Thus, at this point, is reasonable to assume that the loss function actually does not define "the right configuration of $\theta$, but just the "travel" that we do to get there.
Now, my question. Does the loss even care at this point? Because we have for sure that:
$$
MAE_{\theta}(f(x),y) = 0 \iff MSE_{\theta}(f(x),y) = 0 \iff BCE_{\theta}(f(x),y) = 0
$$
thus the "global minima" are for sure shared between those functions.
So my question, saying "does it even care", mean something like:
ok we have shared global minima, so using one or the other, does it give us any other ice property?
In a sense like, do we have a smoother function with one loss or the other? Can we assume to have "better local minima" with one or the other? Do we get better generalization with one or the other (without other interventions like regularization/dropout etc)
(Yes I know the "BCE" helps for the saturated sigmoid section wrt MSE and MAE)

The precise form of the loss function can be important. For example, comparing mean absolute error (MAE) and mean squared error (MSE), the MSE will multiply the error value with itself, in effect giving higher weight to larger errors. This means that if you have one method that gives you very small errors for most observations but large errors for a very few, regarding MAE this will normally be better than a method that gives moderate errors for all observations, whereas regarding MSE it will normally be worse (of course this depends on the precise values).
Whether we want one or another will depend on the specific application. In some applications most if not all observations with very large errors are unreliable outliers and we don't want them to dominate the loss, whereas in other applications moderate errors can be tolerated but large errors can lead to disaster and should be avoided at all costs.
(PS: This is for continuous data not binary; for binary data the discussion is slightly different, but ultimately also there it amounts to understanding what the loss function actually implies, and how this relates to what we want in the application at hand.)
