When should I use the delta method rather than the parametric bootstrap? Suppose I am willing to assume a particular likelihood function for an applied statistics problem. I am able to derive the MLE for the parameter $\theta$, which I will call $\hat{\theta}$. I can also derive a standard error for $\hat{\theta}$.
I am also interested in generating a point estimate and confidence interval for some transformed parameter $\tau = g(\theta)$. A statistics textbook might remind me that $\hat{\tau} = g(\hat{\theta})$ is the MLE for $\tau$ due to the equivariant property of the MLE and that I can use the delta method to get standard errors for $\tau$.
I can also use the parametric bootstrap to get confidence intervals for $\hat{\tau}$ (and, for that matter, $\hat{\theta}$), since I am assuming a parametric model.
I have a suspicion that the parametric bootstrap is a "safer" way to form confidence intervals than the delta method, but that the delta method is emphasized in statistics classes because it is a nifty mathematical result and because of historical reasons (for example, statistical instruction predates fast computers). Downsides to the delta method are:

*

*CIs based on a normal approximation often don't work because asymptotic normality hasn't "kicked in" yet

*It may be hard to simply time consuming to derive

*Even when it is tractable, it's easy to make math mistakes in deriving the standard error

My question is: in practical applications (rather than when we are doing school or theory), do people really find the delta method more useful than the parametric bootstrap in many cases? Or should I think of the parametric bootstrap as my "default" method for computing CIs in this situation (with the possibility of overruling this default, if there are special circumstances to the problem at hand)? Or perhaps these questions are wrong-headed for some reason I haven't considered?

People really do find the delta method more useful in some cases. It is really impossible to tell which is more appropriate without being given a specific context since the context would largely determine which method is "best" for the given situation. I think that you've considered this to at least some extent, because you've listed some downsides to the delta method that you came up with.
I pick up on some skepticism, or sentiment that the delta method may be favored by academics without being pragmatic in real-world applications where we have modern computational statistics. The skepticism is good, but consider your downsides to the delta method. The first only holds when asymptotic normality hasn't "kicked in" as you put it, but it can "kick in" faster than you'd expect in some applications. Mistakes in the derivation are possible, but not a serious concern since you'd typically not be doing anything wildly different from one application to another. And it usually isn't too time-consuming to derive. If you're willing to use a computer for bootstrapping, then you can use a computer for the delta-method... and I guarantee that it will take far less computation than bootstrapping.
Here's a real-world example of the delta-method that is often used in reality: Greenwood's formula. Greenwood's formula uses the delta-method to come up with standard errors for Kaplan-Meier curves, which are used in survival analysis (medical studies, mostly). This is just one very big example off the top of my head that uses the delta method in a major real-world application, and I'm sure there are more.
I think you can answer your own question if you give yourself a context. Just make pros and cons for bootstrapping and the delta-method like you already did with the delta-method, and ask yourself "for my given situation, which method's pros outweigh the cons?"
