Can I make a statement about how much larger of a delta a metric needs to be, such that a neutral conclusion become a significant one? Apologies for the convoluted title. Couldn't figure out a more elegant 1 sentence description.
Here are the details.
I have an experiment measuring a metric e.g. conversion rate, that concluded with a confidence interval straddling 0, and a p-value larger than the classic 0.05 threshold. This points to insufficient evidence to reject the null, a scenario I'm describing as a "neutral" result.
With the information I have from this experiment, is there anything wrong with me making the following claim?

*

*I can calculate the standard error from the confidence interval e.g. CI of [-0.5, 1.3] = SE of 0.9

*Can I claim that had this experiment resulted in an observed delta of 0.9, assuming S.E remains the same, my new C.I would be [0.0, 1.8] leading to a conclusion of statistical significance?

My instincts tell me there's a big flaw in this reasoning but I can't figure out where exactly.

Rather than theorizing what if the estimated effect were twice as big, you can plan the next experiment so that its standard error is twice as small. (For the experiment that's already over, there is nothing to do other than present the results clearly and truthfully.)
To make this concrete, let's suppose that you are working with a confidence interval of the form:
$$
\begin{aligned}
\bar{x} \pm z_{\alpha}\frac{\sigma}{\sqrt{n}}
\end{aligned}
$$
where $\bar{x}$ is the estimate and

*

*$z_{\alpha}$ is the critical value, $\sigma$ is the standard deviation, $n$ is the sample size

*$\sigma/\sqrt{n}$ is the standard error;

*$z_{\alpha}\sigma/\sqrt{n}$ is the margin of error.

Here $z_{\alpha}$ is determined by the significance level $(1-\alpha)$100% and $\sigma$ is a property of the population under study. You probably want to keep $\alpha$ and $z_{\alpha}$ fixed, and it's hard to change $\sigma$.
But you do get to choose the sample size $n$. You can make the margin of error (the half-width of the confidence interval) as small as you want by increasing $n$. In your case, $\bar{x} = 0.4$ and $z_{\alpha}\sigma/\sqrt{n} = 0.9$, so the margin of error is more than 2 × the estimate; the sample wasn't large enough to estimate the effect very precisely.
And here is where planning for the next experiment comes in. If an effect size of 0.3 is the smallest effect size of practical interest, you'd want the margin of error to be less than 0.3. So you use the results from the concluded experiment to get an estimate $\hat{\sigma}$ of the standard deviation $\sigma$ and then you solve for $n$ in the formula $z_{\alpha}\hat{\sigma}/\sqrt{n} < 0.3$.
PS: I deliberately choose the "smallest valuable" effect to be smaller than the effect observed in the pilot study. This is the effect that you would regret to miss as "non-significant" because the followup experiment wasn't large enough to detect it at the $(1-\alpha)$100% level.
