What happens if we follow the gradient of a softmax activation Given a softmax output layer, what does it mean to "follow the gradient"? Usually that would consist in "increasing the output" but obviously the softmax has no notion of "increasing the output" since it's normalized
So, I was wondering what actually happens following the gradient of a Softmax layer, and my take is that the output stays the same, but the pre-activations increases, thus having
$$
pre = W*x+b\\
o = softmax(pre) 
$$
will increase the values of $pre$

Usually, neural networks use gradient descent, not ascent, because there is a loss function to be minimized. But, this doesn't matter for the discussion, as   the same problem can be formulated as maximization of negative loss, or likelihood. So, let's assume we use gradient ascent, call the loss function as the optimization function.
The final gradients we care are always the optimization function's gradients with respect to the network parameters'. Here, $f$, is the optimization function. With gradient ascent, we expect to increase the value of $f$. But, that doesn't mean we increase the intermediate layers' outputs inside it. It's the final value of the loss/optimization function that's being affected from the ascent or descent strategy.
For example, if the output label is $y=1$, and the loss function is $f=(y-o)^2$, and we use gradient descent, the output, $o$, should get close to $1$ to decrease the loss. This means, with gradient descent, we decrease the loss but increase the network's output for this test case.
