matrix-calculus - Understanding numerator/denominator layouts Consider the following machine-learning model:


Here, $J = \frac{1}{m} \sum_{i = 1}^{m} L(\hat{y}^{(i)}, y^{(i)})$, and $m$ is the number of training-examples.

While performing reverse-mode differentiation (or back-propagation), I have the following questions:

Using the numerator-layout.


*

*What would be the dimension of the derivative $\frac{\mathrm{d} J}{\mathrm{d} \mathbf{L}}$?

*

*Should it be a column-vector of dimension $(m, 1)$, because $\mathbf{L}$ is a row-vector of dimension $(1, m)$ (Source: here)

*

*However, using this notation causes issues while computing the derivative $\frac{\mathrm{d} J}{\mathrm{d} \mathbf{a}} = \frac{\mathrm{d} J}{\mathrm{d} \mathbf{L}} \frac{\mathrm{d} \mathbf{L}}{\mathrm{d} \mathbf{a}}$; since, $\frac{\mathrm{d} \mathbf{L}}{\mathrm{d} \mathbf{a}}$ would be an $(m, m)$ matrix, while $\frac{\mathrm{d} J}{\mathrm{d} \mathbf{L}}$ is an $(m, 1)$ vector.

*But, this notation does serve well when computing the derivatives of the form $\frac{\mathrm{d}y}{\mathrm{d}\mathbf{X}}$, where $y = f(X)$; $\mathbf{X}$ is a matrix of dimension $(m, n)$; and $f(\mathbf{X})$ is a scalar-valued function.



*Or should it be a row-vector, because according to the numerator-layout the derivative has the dimensions --> $\text{numerator-dimension} \times (\text{denominator-dimension})^\intercal = (1,1)\times(m, 1)$ (Source: here)

*

*Also, (for this point) is my understanding even correct?





PS: also, is there any definitive guide from which I can learn matrix-calculus from the first principals. Although, the following sources are good, they still leave a lot of gaps:

*

*Matrix-Calculus

*The Matrix-cookbook

*Old and New Matrix Algebra Useful for Statistics, by T. P. Minka

*The Matrix Calculus You Need For Deep Learning

Unfortunately, I didn't come across a resource that doesn't leave gaps. It's a disputed area. Even the chain rule may sometimes not make a lot sense, e.g. some terms might be 3D tensors that the matrix multiplication is not well-defined because of matrices differentiated by vectors or vice versa.
Having said that, these rules are also very useful if you comply. For ex, define your vectors as column vectors, i.e. $n\times 1$. The chain rule looks like:
$$\frac{\partial J}{\partial \mathbf a^T} = \frac{\partial J}{\partial \mathbf L^T}\frac{\partial \mathbf L^T}{\partial \mathbf a^T}$$
where all the row vectors are transposed. Then, the matrix multiplication sizes would match. All equations you see in the sources assume a harmony, e.g. a chain rule expanding right as above would not make sense with denominator notation, so it'd expand left instead, but the sources rarely mention it. Therefore, whenever a vector is of concern, it'd be better to assume it's a column vector (wikipedia equations note this).
