How well do power calculations actually work in reality? I'm planning a study and I need to do a power calculation. So much of it seems to depend on guesswork and estimations of what the data (that I don't have yet) will look like. Guesswork seems to be such standard practice I can't help but wonder how accurate they are in practice. Does anyone know of a study that's compared predicted power against association rate?
e.g. of all the studies claiming they will 90% power to detect an association at p<0.05, what percentage actually find an association at p<0.05 when they get the data and that prediction hits the real world? Is it close to 90?
This is purely curiosity driven, I understand the need for a power calculation, I'm just curious how well they actually work across science.

To answer this question, it is useful to first separate the power function itself from sample-size calculations that try to achieve a stipulated level of power against an alternative hypothesis.  The power function arises whenever we have a proposed method of hypothesis testing and it measures the probability of rejecting the null hypothesis in the test, conditional on the true value of the parameter under analysis.  The power function is a function of the parameter of interest in the test, the significance level for the test, and the sample size.  However, it does not depend on the data values --- just the sample size.
Now, when we undertake sample-size calculations on the basis of a power requirement, we need to stipulate three things: (1) the significance level for the test; (2) the minimum level of power we want to achieve; and (3) the (alternative) parameter value against which we wish to achieve this level of power.  Once we stipulate these three things, there is some minimum sample size $n$ that will achieve the required level of power against the stipulate value of the parameter.  There is no guesswork of what the data will look like here, because the content of the data are not required for this computation.
As to the idea of undertaking empirical tests to do a post hoc check of the power,  the only thing that is really worth checking is whether or not the underlying model assumptions for the test are accurate or not.  If the underlying model assumptions are accurate then the power calculation is accurate --- it merely reflects a mathematical property of the test under that model.  If an empirical assessment of the power curve were to examine the rejection rate in studies, it would need to know  the true parameter values in those studies to know where on the power curve it is supposed to be comparing the empirical rejection rates to.
