Linear Classifiers -- Single Linear Layer with k Neurons vs 'one vs rest' (k Linear Layers with 1 Neuron) When using a linear classifier for a k-class classification problem, is there actually any difference of using

*

*a single linear layer with k neurons + cross entropy loss

*'one vs rest': k linear layers with 1 neuron each + binary cross entropy loss [*]

In particular, should one be preferred over the other? Is there any crucial difference wrt  convergence or convergence time? I apologize if this question seems very 'basic'.
[*] Detailed explanation: For this approach, k neurons are trained, one for each class and thus, the problem is reduced to k binary classification problems. For each of the neurons, either an observation belongs to the class the neuron is assigned to ('one') or not ('rest'). During inference time, the class of an observation is determined by feeding the observation into all k neurons and assigning the class of the neuron that produces the highest class score (simply corresponds to an argmax over the class scores).

In the first option, each neuron already follows a one vs. rest approach because its output is supposed to be closer to $1$ if only a sample from the corresponding class is given, and for other classes, it's expected to be closer to $0$. The decision is again based on the neuron with maximum output. So, assuming all neurons have the same output, e.g. sigmoid, architecture-wise, the two approaches are actually equivalent. It doesn't matter if they're trained separately or not because gradients do not clash.
But, their loss functions differ. In the first one, for a sample with class label $u$, the loss function is $-\log p_u$, where $p_u$ is the $u$-th neuron's output. In the second one, it is $-\log p_u - \sum_{i\neq u}\log (1-p_i)$. So, there is an extra term in the loss coming from the other neurons. You could have used the same loss function for the first architecture. So, it's not a matter of which arch. is useful or not, but a matter of loss function.
Unfortunately, there is no principled way to say which one is better. It depends on the problem, on the data, and the best way to find out is through validation. But, usually, another option is more common; and that is using a softmax at the end together with multi-class cross-entropy loss, which play along well in gradient-based opt. approaches.
