Disadvantages of using a regression loss function in multi-class classification Given $k > 2$ classes, consider the following loss function
$$
\sum_i||y^{(i)} - \hat y^{(i)}||^2
$$
Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,1]^k$ is the prediction (obtained after applying a sigmoid to the output logits of some neural network).
What are the theoretical and practical disadvantages of using such a loss over something like the cross-entropy loss?

The cross-entropy loss gives you the maximum likelihood estimate (MLE), i.e. if you find the minimum of cross-entropy loss you have found the model (from the family of models you consider) that gives the largest probability to your training data; no other model from your family gives more probability to your training data. (A model family might be e.g. the set of all possible weight assignments to some chosen neural network design.)
Being an MLE helps with mathematical reasoning about the properties of your result because there is lots of theory for MLEs.
Also, cross-entropy would be a little faster to compute than the sum of squared error (SSE) loss you mention.
Some people argue, that SSE loss is inferior because the loss depends not only on the probability of the correct label under your model but also on the distribution of the probabilities that the model gives to the wrong models (since it is not linear).
But, as far as deep neural networks are concerned, the real reason why cross-entropy is used most often (not always) for those, is that experience shows that it is very often leading to better results. We just haven't found something truly better yet (that would also be practical).
