Hi,
There is something else in this context which has been bothering me–
Suppose we have an input layer X0, weight W1, and output X1, then


X1.gradient is a scalar formed by summing up the gradients of all units of X1


W1 is a weight matrix containing internal weights w(i,j) which are updated based on :
* The scalar gradient of X1
* And the vector X0


This however, brings me up to the conclusion that for each w(i,j) to be updated individually, we should have the scalar X1 and each unit of the X0 matrix available for the specific i and j – Which is not plausible as the X0 matrix has a different shape as compared to the W1 matrix. How exactly are individual weights of the W1 matrix computed?