p-values from t.test and prop.test differ considerably Testing a difference in hits (ones versus zeros) between two independent groups $X$ and $Y$ should be possible with a t-test, according to the following considerations:

*

*$x_i\in\{0,1\}$ is the measurement for the $i$-th item im group $X$, and $y_i\in\{0,1\}$ the same for group $Y$

*the proportion in each group is the mean value of the measurements, i.e. $\mu_X=\sum_i x_i/n$ and $\mu_Y=\sum_i y_i/n$

*the difference of mean values $\mu_X$ and $\mu_Y$ in two groups can be tested with a t-test

In this particular case, a proportion test (function prop.test in R) is an alternative test option. Interestingly, the results are quite different:
> x <- c(rep(1, 10), rep(0, 90))
> y <- c(rep(1, 20), rep(0, 80))
> t.test(x,y,paired=FALSE)
t = -1.99, df = 183.61, p-value = 0.04808
> prop.test(c(10,20), c(100,100))
X-squared = 3.1765, df = 1, p-value = 0.07471

Note the higher p-value of the prop.test. Does this mean that a t-test has a higher power, i.e., can distinguish between $H_0$ and its alternative already for smaller $n$? Is there a reason why a t-test should not be used in this case?
Addition (Edit: resolved in a comment under the answer by Thomas Lumley below): The result of the t-test is even more surprising in the light of the observation that even the asymptotic ("Wald") 95% confidence intervals of both measurements overlap (0.1587989 > 0.1216014):
> library(binom)
> binom.confint(10, 100, method="asymptotic")
      method  x   n mean      lower     upper
1 asymptotic 10 100  0.1 0.04120108 0.1587989
> binom.confint(20, 100, method="asymptotic")
      method  x   n mean     lower     upper
1 asymptotic 20 100  0.2 0.1216014 0.2783986

As confidence intervals based on the t-distribution should be even wider than those based on the normal distribution (i.e. $z_{1-\alpha/2}$), I do not understand why the t-test reports a significant difference at the 5% level.

*

*A difference between p-values of 0.048 and 0.074 is not large. This can easily happen between tests that don't do exactly the same but a similar thing.


*The theory of the t-test is for normally distributed data, which your data obviously are not. You're right that the t-test can be justified as an approximation, but there's no reason to use an approximation if a more precise test (namely the proportion test) is available. For sure there is no reason to expect the t-test to have a better power, or only in case that it is anticonservative, which is not a good thing (being an approximation, one would probably need to simulate what its finite sample characteristics are in this situation).


*Edited after looking up the reference Agostino et al. ("The Appropriateness of Some Common Procedures for Testing the Equality of Two Independent Binomial Populations", Am. Stat. 1988) given by cdalitz. This reference states that prop.test with continuity correction is too conservative whereas the t-test as well as the prop.test without continuity correction are normally closer to the nominal level, if occasionally anticonservative (which in may view not necessarily justifies an overall recommendation). This was also mentioned in the answer by Thomas Lumley.
If we're ignoring the continuity correction for a moment, there are two differences between the t-test and prop.test (which is not fully documented but I think it does the z-test based on normal approximation).
(a) prop.test uses the knowledge that the variance of the Binomial is $np(1-p)$ rather than using a sample variance based on normality. In my view what prop.test does here should clearly do better, as it is based on information about the specific setup used here.
(b) prop.test uses a normal approximation whereas the t-test uses a t-approximation. Now both of these, applied to the Binomial situation, are asymptotic in nature (the t-distribution is only precise if the underlying data are normal which they aren't here), and actually they are asymptotically equivalent. Although the normal approximation looks more intuitive based on the Central Limit Theorem, this doesn't imply by any means than the normal works better than the t in the finite sample situation (and the t is as well justified by the CLT, if only indirectly). The t-distribution is motivated by the normal assumption, but in fact it may also be the case that the asymptotic normal distribution of prop.test underestimates the finite sample variability because it ignores the variability in the variance estimation, and the t-distribution, despite here not precisely justified, may do a better job at that.
So I now believe that potentially (as could be confirmed by simulations, maybe somebody has done that?) the best thing to do could be using the test statistic of prop.test, i.e., the "correct" variance estimation, but replacing the asymptotic normal distribution by a t-distribution, which in some sense may put together the advantages of them both.
