All great and valid answers here, and they have helped me see the key piece of the puzzle I was missing: the subtle point that although the instantaneous axis of rotation has zero velocity, it is accelerating upward (centripetal acceleration). Therefore to apply <PERSON>'s second law (or <PERSON>'s law) about this axis, we need to take into consideration the inertial forces/torques that result.
Let $\vec{R}$ denote the position of a point on the instantaneous axis of rotation, relative to a fixed point that coincides with the axis of rotation at $t=t'$, but does not accelerate like it. This means that at $t=t'$, $\vec{R}=0$ and $\dot{\vec{R}}=0$ but $\ddot{\vec{R}}\ne 0$. Let $\vec{r}$ be the position of a particle of $M$ in the rotating system, relative to $\vec{R}$. 766
Its angular momentum about the fixed point is $$\vec{L}=M(\vec R + \vec r)\times (\dot{\vec R} + \dot{\vec r})$$ $$\vec \tau=\frac{d\vec L}{dt}=M(\vec R + \vec r)\times (\ddot{\vec R} + \ddot{\vec r}) $$ At $t=t'$, since $\vec{R}=0$, $$\frac{d\vec{L}'}{dt}=M\vec{r}\times\ddot{\vec r}=\vec{\tau}-M\vec r\times\ddot {\vec R}$$ where $\vec{L}'$ here is the angular momentum about the instantaneous axis of rotation. The second term on the right hand side is the inertial torque that must be added to the torque due to the real forces: $$\vec{\tau}_i=-M\vec{r}\times\ddot{\vec{R}}.$$ $$\ddot{\vec R}=\omega^2R\hat{y}$$ where $\hat y$ is in the upward direction. The intertial torque on mass $m$ in this problem is $\tau_i=m\omega^2R^2\sin\theta$ (clockwise/inward).
All this so far was to derive the expression for the inertial torque. Now writing the second law, with the moment of inertia $I$ taken to be constant, as in case 1 in the question: $$mgR\sin\theta + m\omega^2R^2\sin\theta= \tau'= I\alpha $$ $$\alpha = \frac{g}{2R} \sin\theta \frac{1 + \frac{\omega^2}{gR}}{1 + \cos\theta} $$ Plugging in $\omega^2$ leads to the correct expression: $$\alpha = \frac{g}{R}\sin\theta\frac{1}{(1 + \cos\theta)^2}. $$ 804