Does Fisher scoring exist as such? I'm studying second order optimization methods in statistics, and I've run into a conceptual barrier that I was hoping someone can help me with.
First for some notation: consider a sample $X_1,\dots,X_n\thicksim P_{\theta^\star}$ [edit: $\theta^\star$ is the true, unknown parameter. I had originally denoted it as $\theta_0$, which has been misunderstood as the first of the iterates $\theta_k$ discussed below -- sorry for the poor notation], and our goal is to find the MLE $$\hat\theta=\operatorname{arg\,max}_{\theta}L(\theta)=\operatorname{arg\,max}_{\theta}\sum_{i=1}^n\ell(X_i, \theta)=\operatorname{arg\,max}_{\theta}\sum_{i=1}^n\log P_{\theta}(X_i)$$.
We consider two iterative algorithms:

*

*Newton-Raphson: $\theta_{k+1}=\theta_k + (-\nabla^2 L(\theta_k))^{-1}\nabla L(\theta_k)$

*Fisher scoring/natural gradient: $\theta_{k+1}=\theta_k + (I(\theta_k))^{-1}\nabla L(\theta_k)$, where under the usual assumptions $$I(\theta)=E_{\theta^\star}[-\nabla^2 L(\theta)].$$
[edit:the expectation in this formula must be over $\theta$ instead of over $\theta^\star$, this misunderstanding was at the core of my confusion.]
Now, it seems to me that: since we don't know $\theta^\star$ (and therefore expectations under $P_{\theta^\star}$ can't be computed), it follows that to implement Fisher scoring we must approximate $I(\theta_k)$ with some approximation, for instance $(\nabla L(\theta_k))^\top(\nabla L(\theta_k))$ or $-\nabla^2 L(\theta_k)$. The first approximation is reminiscent of the Gauss-Newton algorithm, and the second is exactly Newton-Raphson.
Assuming what I have written above is correct/makes sense, my question is: am I correct that in implementation, Fisher scoring is always carried out by an approximation like the above? Is it ever possible to carry out "exact" Fisher scoring using the expected information matrix? I've read a lot of sources where these algorithms are presented separately, so I'm uncertain, and I'd appreciate any clarification. Sorry if this is a duplicate -- there are lots of similar questions here which I have read, but I could not find one that addressed my question.

As mentioned before, you start with a random value of $\theta_0$, and then you (hopefully) CAN calculate the expectation. So the expectation is w.r.t. the current parameter value. Without a starting value you wouldn't be able to calculate the gradient either.
You considered Fisher-Scoring and Natural Gradient as equivalent. While they are quite similar, note that in Fisher-Scoring, the objective is always the (log) likelihood function - that is the log of some distribution function. In Natural Gradient the objective can be what-ever function as long as you are taking the gradient w.r.t. to the parameters of some distribution that is part of this function. Note that also the motivation in both algorithms were different. Still, I think you could say Fisher-Scoring is a special case of Natural Gradient (though I'm not an expert on the topic, and there might be some subtleties that elude me).
