How well do power calculations actually work in reality? I'm planning a study and I need to do a power calculation. So much of it seems to depend on guesswork and estimations of what the data (that I don't have yet) will look like. Guesswork seems to be such standard practice I can't help but wonder how accurate they are in practice. Does anyone know of a study that's compared predicted power against association rate?
e.g. of all the studies claiming they will 90% power to detect an association at p<0.05, what percentage actually find an association at p<0.05 when they get the data and that prediction hits the real world? Is it close to 90?
This is purely curiosity driven, I understand the need for a power calculation, I'm just curious how well they actually work across science.

*

*You don't know when $H_0$ is false, so you can't compute the correct empirical rejection rate for power from looking at the results of tests. You shouldn't include the cases where the null was true in that calculation.  (If you could say when $H_0$ was false, you wouldn't need tests in the first place.)
Further, even if you could tell when $H_0$ was true or false, it would still be wrong.  Unless you worked with a whole power curve across effect sizes, desired power was specified at some given population effect size. The sample equivalent is the proportion of correct rejections at that population effect size, so the denominator would be limited to the number of comparable cases where the effect was actually that size (which of course you don't know).


*That said, power calculations are based on your requirements in relation to the population -- what's the minimum power you require at some meaningful  population effect size -- e.g. it might be an effect that is clinically important or educationally relevant (e.g. the power you want if the effect was a 5% higher pass rate on some skills test, say), and so forth -- an amount you would be interested in having good power at.
What is an effect size of interest does not typically come from data. It is not a statistical consideration, but a subject matter one.  To this end it's usually better (where at all possible) to frame such considerations in terms of raw effects, not standardized effects, since it's on the scale of raw effects that such considerations would typically apply. It is also not a matter of a few moments thought (for all that this often seems to be all it's accorded from what I have seen); it also requires solid domain knowledge to understand what such an effect size might be.


*There's a tendency I've seen to base power calculation inputs on the estimated effects from some previous study. There's often no obvious reason why a noisy estimate of a population effect size should be an effect size at which you want some given power -- this is  conflating two distinct tasks, with distinct considerations.
However, if one were to use estimates of quantities like (say) an estimate of a population mean and standard deviation, it would be quite wrong to treat those as if they were the population values. This practice will lead to typically lower power than the usual (population-based) calculations suggest, and sometimes much less. One appropriate way to investigate this (particularly for more complicated models) would be via simulation.
