I’m trying to understand how to use the gradient of softmax.
For example, if I had an input x = [1,2] to a Sigmoid activation instead (let’s call it SIG), the forward pass would return the vector [1/1+e^1, 1/1+e^2] and the backward pass would return gradSIG/x = [dSIG/dx1, dSIG/dx2] = [SIG(1)(1-SIG(1)), SIG(2)(1-SIG(2))]. That is, the gradient of Sigmoid with respect to x has the same number of components as x.
But with the softmax (let’s call it SMAX), the gradient is usually defined as SMAX(i)*(1-SMAX(j)) if i = j, else -SMAX(i) * SMAX(j). I understand this as meaning that softmax returns the full Jacobian matrix with:
[[dSMAX(x_1)/dx_1, dSMAX(x_1)/dx_2], [dSMAX(x_2)/dx_1, dSMAX(x_2)/dx_2]].
These are 4 components instead of 2 but x has 2 components only. So should I be summing dSMAX(x_1)/dx_1 + dSMAX(x_2)/dx_1 and dSMAX(x_1)/dx_2 + dSMAX(x_2)/dx_2?