Confidence interval / p-value duality: don't they use different distributions? Idea:

*

*p-value is less than the level of significance if and only if the corresponding CI does not include the null value; and vica versa,

*the p-value is greater than the level of significance if and only if the corresponding CI does include the null value.

Is this idea always correct? Is it only correct for certain sampling distributions, like the normal, but not correct in general?
If the idea is correct in general, then why? The p-value is calculated using the distribution of the statistic conditional on H0, while the CI is calculated using the unconditional distribution of the statistic. These are two different distributions - how or why does this lead to the duality?
If the idea is not correct in general, could you provide counterexamples? Do people typically think that these counterexamples are a problem and somehow try to correct them, to make the idea still be true?

Yes, both formulations are identical because this is just the definition of a confidence interval. Formally, if you have measured the parameter estimate $\theta_0$ for the parameter $\theta$, a $1-\alpha$ confidence interval $[\theta_1,\theta_2]$ is given by
\begin{eqnarray*}& & P(\hat{\theta} \geq \theta_0|\theta=\theta_1)=\alpha/2 \\
\mbox{and}\quad & & P(\hat{\theta}\leq \theta_0|\theta=\theta_2)=\alpha/2
\end{eqnarray*}
This definition distributes the error probability equally on both sides, so it is only equivalent to a two-sided test. Compared to the hypothesis testing scenario, the condition in the probability is the null hypothesis, which means that $[\theta_1,\theta_2]$ are just the borders of the rejection region.
Remark: Interestingly, most textbooks on statistics give a different defintion of a confidence interval: $P(\theta\in[\theta_1,\theta_2])=1-\alpha$. This definition depends on the unknown parameter value $\theta$ and only allows for solving for $\theta_{1/2}$ in very special cases: the asymptotic normal aproximation ("Wald interval") for the binomial proportion (which is problematic, as pointed out in other answers) and the confidence interval for a statsitical mean. I have found the general definition in

DiCiccio, Efron: "Bootstrap confidence intervals." Statistical Science, pp. 189-228, 1996

