2nd order Taylor expansion of KL divergence I am having trouble understanding page 6 of this PDF: https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/lecture-notes/lecture20-2pp.pdf
This is also a question in cs229-autumn2018 PS#3-3d.
In particular, page 6:
$$
D_{KL} \big( P(X;\theta)||P(X;\theta+d) \big) \\
= \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;\theta+d)} \\
\approx \Sigma_x P(x;\theta) \Big(
\frac{P(x;\theta)}{P(x;\theta)} 
- d^T \nabla_{\theta} lnP(x;\theta)
- \frac{1}{2} d^T \nabla^2_{\theta} lnP(x;\theta) d
\Big)
$$
This is where I am having an issue. The approximation is a 2nd order Taylor expansion of $D_{KL}$, but instead of variable $x$ or $\theta$, we have a function $P(x)$ parameterized by $\theta$. I do not understand how this works.

To further illustrate where exactly my confusion lies, my intuition is that in normal Taylor expansion, e.g. $f(x)$ at $a=5$, our expansion would simply be
$$\frac{1}{0!} f(5) (x-5)^0 + \frac{1}{1!} \frac{df}{dx}(5) (x-5)^1 + \frac{1}{2!} \frac{d^2f}{dx^2}(5) (x-5)^2$$
Now, for $D_{KL}$, we treat $(\theta+d)$ as a variable and $\theta$ as a constant. We take the derivative w.r.t to $(\theta+d)$. For where we would replace $x$ with $a=5$ in normal Taylor expansion, we replace $P(\theta+d)$ with $P(\theta)$.
For simplicity's sake, we denote $(\theta + d)$ as $t$. Thereby, we get:
$$
D_{KL} \big( P(X;\theta)||P(X;t) \big) \\
= \Sigma_x P(x;\theta) ln\frac{P(x;\theta)}{P(x;t)} \\
$$
The first term in the expansion is simple, no derivatives are involved. We simply replace $t$ with $\theta$.
$$ln\frac{P(x;\theta)}{P(x;\theta)}$$
The second term, ignoring the $(x-a)$ part, is
$$
\nabla_t ln\frac{P(x;\theta)}{P(x;t)} \\
= \nabla_t lnP(x;\theta) - \nabla_t lnP(x;t) \\
= 0 - \nabla_t lnP(x;t) \\
= \nabla_{\theta} lnP(x;\theta)
$$
And baam... How is $\nabla_t lnP(x;t)$ supposed to be equal to $\nabla_{\theta} lnP(x;\theta)$?
If so, why doesn't
$$\nabla_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$
or
$$\nabla^2_t ln\frac{P(x;\theta)}{P(x;t)} = \nabla^2_{\theta} ln\frac{P(x;\theta)}{P(x;\theta)} = 0$$
which the second one is clearly wrong?
I felt I am missing some fundamental concepts here. Thank you for your help.

I believe this: Information Geometry and Natural
Gradients  --- by Nathan Ratlif can answer your question. Please find the detailed derivation in Section 3 on Page 6-8. I just copy the part below.



