<p>I agree with @AlecosPapadopoulos we want something like:
<span class="math-container">$$ \Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$</span>
But I don't see how <span class="math-container">$n$</span> and <span class="math-container">$M$</span> couldn't enter into the proper test statistic.
For example, if the underlying data is normally distributed i.i.d. data then  <span class="math-container">$N$</span> and <span class="math-container">$M$</span> do matter.</p>
<p>Consider that noise mean <span class="math-container">$\mu$</span> and variance <span class="math-container">$\sigma$</span>, which, by assumption is the same for the control and &quot;treated&quot; group.  The mean of the treated group with size N will be distributed <span class="math-container">$N(\mu, \sigma^2 / n)$</span> and <span class="math-container">$N(\mu, \sigma^2 / M)$</span> for the control. So the difference in means will be distributed <span class="math-container">$$N(0, \sigma^2 / n + \sigma^2 / m)$$</span></p>
<p>But you won't know <span class="math-container">$\sigma$</span> or <span class="math-container">$\mu$</span>, so we'll have to estimate it with <span class="math-container">$X_1$</span>, <span class="math-container">$X_2$</span>, and <span class="math-container">$s_{X_1X_2}$</span>, and use a t-test. This setup gives a t-statistic like this:
<span class="math-container">$$t = \frac{\bar {X}_1 - \bar{X}_2}{s_{X_1 X_2} \cdot \sqrt{\frac{1}{n}+\frac{1}{m}}}$$</span>
where
<span class="math-container">$$s_{X_1X_2} = \sqrt{\frac{(n-1)s_{X_1}^2+(m-1)s_{X_2}^2}{n+m}}.$$</span>
<a href="https://en.wikipedia.org/wiki/Student%27s_t-test" rel="nofollow noreferrer">SRC: Student's t-test on Wikipedia</a></p>
<p>The unpaired sample t-test for this difference in means has degrees of freedom <span class="math-container">$N-M-2$</span>. Therefore the rejection region should depend on both n and m, both in what critical value of the test to use through the degrees of freedom of that test and the test statistic calculation itself.</p>

CC BY-SA 3.0
2015-05-31T12:30:15.110Z
Certainly the sample sizes enter the test-statistic, but this does not affect the specific probability the OP asks about (i.e. the a priori probability that we would obtain at least one p-value lower than a given threshold).
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-31T13:31:56.520Z
A properly specified test would incorporate the multiple comparisons just as it would sample size and degrees of freedom - - right? So is this a question about using the wrong test statistic along only the multiple comparison dimension but properly specified on the individual comparison dimension? Because the presence of m and n in the test statistic is exactly because a small sample size is more likely to have a large difference under the null hypothesis.
BKay
CC BY-SA 3.0
2015-05-31T13:55:51.840Z
The crucial point here is that a priori, a) a p-value has a marginal U(0,1) distribution, _irrespective of anything else_ (sample size or whatever). This is a general result holding in any kind of situation. This should be intuitive: what would happen if the p-value had a priori a non-uniform distribution? CONTD
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-31T13:59:20.387Z
CONTD  It would mean that some of its values would be more likely than others, again _irrespective of the specifics of the situation_. But this would invalidate  the whole testing procedure, _exactly because_ it would imply that "it doesn't matter what you test, what  is the sample size etc - _this_ value for the p-value is more probable than others".
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-31T16:32:21.843Z
Thank both of you for the extended discussion, I need to learn more about this and I feel this should definitively be part of a graduate studies curriculum.
FooBar
CC BY-SA 3.0
2015-06-01T13:41:48.723Z
@FooBar It usually is in Science/Engineering. We got taught explicitly that p < 0.05 wasn't acceptable.
Lumi
CC BY-SA 3.0
2015-06-01T23:59:16.493Z
@Lumi "p < 0.05 isn't acceptable" : not acceptable for what purpose? What were you taught that it would be "acceptable"?
Alecos Papadopoulos
CC BY-SA 3.0
2015-06-02T16:47:27.093Z
@AlecosPapadopoulos Essentially not acceptable to justify publishing without  a strong theoretical basis as well. In fact, he wasn't particularly comfortable even with p < 0.01 just on its own, unless it was a very well designed experiment. This was a very good statistics lecturer who was already quite horrified at the way easy to use computer based statistical packages were essentially enabling p-hacking type approaches.  The run all the statistical tests until you get the magic number approach trap that students quite easily fall into.
Lumi
CC BY-SA 3.0
2015-06-02T17:57:52.277Z
@Lumi That is a wise approach. My approach to p-values is that they are just one more piece of evidence, not the critical or the leading one.
Alecos Papadopoulos
CC BY-SA 4.0
2015-05-31T11:12:15.277Z
2021-11-23T12:41:04.810Z
2021-11-23T12:41:04.810Z
BKay
<p>Under the assumption of i.i.d. Normal  characteristics, the situation described is taken care by separate <a href="https://en.wikipedia.org/wiki/Welch%27s_t_test" rel="nofollow noreferrer">Welch's t-tests</a> that account for possibly different sample sizes and different variances. Denote the statistics of these tests <span class="math-container">$t_j, j=1,...,K$</span>.
The p-value associated with each is</p>
<p><span class="math-container">$$p_j = \Pr\big(|t_j|\geq t(\alpha)\mid H_0\big)  $$</span></p>
<p>where <span class="math-container">$H_0$</span> is the hypothesis that the populations means between treated and controlled group are equal, and <span class="math-container">$t$</span> depends on the significance level <span class="math-container">$1-\alpha$</span>.</p>
<p>We can write the probability in terms of the corresponding  cumulative distribution function,</p>
<p><span class="math-container">$$\Pr\big(|t_j|\geq t(\alpha)\mid H_0\big) = 1 - F(|t_j|)$$</span></p>
<p>Therefore</p>
<p><span class="math-container">$$p_j = 1 - F(|t_j|) \implies 1-p_j = F(|t_j|)$$</span></p>
<p>If we contemplate the situation a priori, before even looking at the data, then the p-values lie in the future and can be modelled as random variables. Viewed as a random variable, the <a href="https://en.wikipedia.org/wiki/Probability_integral_transform" rel="nofollow noreferrer">probability integral transform</a> tells us that <span class="math-container">$1-p_j$</span> follows a <span class="math-container">$U(0,1)$</span> Uniform distribution, and by the properties of this distribution so does <span class="math-container">$p_j$</span>.</p>
<p>Collecting all <span class="math-container">$p_j$</span>, we have a sample of size <span class="math-container">$K$</span> of independent <span class="math-container">$U(0,1)$</span> uniforms. The probability that at least one of them is smaller than a specific value, say <span class="math-container">$p^*$</span>, is equal to the probability that the minimum of them is lower than this threshold. This can be understood as follows:</p>
<p><span class="math-container">$$\Pr\Big (\text {At least one $p_j \leq p^*$} \Big) = \Pr\Big (\text {Not all $p_j &gt; p^*$} \Big) $$</span></p>
<p><span class="math-container">$$ = 1-\Pr\Big (\text {All $p_j &gt; p^*$} \Big) =  1- \prod_{j=1}^K \Pr\Big ( p_j &gt; p^* \Big)$$</span></p>
<p>due to independence, and so, since they are identically distributed,</p>
<p><span class="math-container">$$\Pr\Big (\text {At least one $p_j \leq p^*$} \Big) = 1- \left [1-\Pr\Big ( p \leq p^* \Big)\right]^K = 1 - \left [1-F_U \big(p^* \big)\right]^K$$</span></p>
<p>But this is the cumulative distribution function of the minimum of <span class="math-container">$K$</span> i.i.d random variables.</p>
<p>Denote this minimum <span class="math-container">$p_{(1)}$</span>.</p>
<p>The CDF of the minimum of <span class="math-container">$K$</span> independent <span class="math-container">$U(0,1)$</span> variables is</p>
<p><span class="math-container">$$F_{p_{(1)}}(p_{(1)}) = 1 - \big [1-p_{(1)}\big]^K$$</span></p>
<p>We want the probability</p>
<p><span class="math-container">$$ \Pr(p_{(1)} \leq p^*) = 1- \big [1-p^*\big]^K$$</span></p>
<p>Indicative values:</p>
<p><img src="https://i.sstatic.net/BAnIC.png" alt="enter image description here" /></p>

CC BY-SA 3.0
2015-05-30T23:28:12.417Z
"*Welch's t-tests that account for possibly different sample sizes and different variances*". I had only time to skim the reply so far, but I can't find  where the sample sizes $M$, $N$ enter the picture. How does the last table vary with the sample size?
FooBar
CC BY-SA 3.0
2015-05-30T23:36:22.130Z
It doesn't. As random variables, the p-values are Uniforms (0,1) whatever the other aspects of the situation. The only thing that matters is the size of $K$.
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-30T23:47:38.857Z
That's strange. [This link](http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800) claims something else: "*If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people.*". What are the underlying assumptions here that lead to there different result?
FooBar
CC BY-SA 3.0
2015-05-31T00:07:18.003Z
The crucial part of the statement is "a large number of things", i.e. a large $K$. Go back to the link and read just below the sub-chapter "The Hook". You will see the same formula as in my answer, and a percentage consistent with my table.
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-31T00:27:00.427Z
I see. So his formulation "large number of things about a small number of people" implies the wrong thing, it should be without "small number of people"
FooBar
CC BY-SA 3.0
2015-05-31T00:32:12.803Z
Yes. I would write it as "If you measure a large number of things (about) _even for_ a small number of people, you are almost guaranteed to get a “statistically significant” result." The size of the sample is of course very important, but in relation to other issues.
Alecos Papadopoulos
CC BY-SA 4.0
2015-05-30T23:16:00.943Z
2021-11-22T21:54:41.507Z
2021-11-22T21:54:41.507Z
Alecos Papadopoulos
<p>The other answers are good, but I thought that another answer with a slighlty different focus might be a good complement.</p>

<p><strong>Does the sample size usually affect the false-positive rate?</strong></p>

<p>Judging from the comments, I think that the question has been prompted by <a href="http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800" rel="nofollow">this article</a>, which includes a couple of mistakes (or at least miswordings).</p>

<p>Firstly (and most worryingly in general) it defines p-values incorrectly, but more relevantly it includes the sentence <em>"If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result."</em></p>

<p>The p-value is the probability, assuming that the null hypothesis is true, of observing a result as least as extreme as the one that was actually observed. As pointed out in the other answers, this means that it should be uniformly distributed between 0 and 1 regardless of sample size, underlying distributions, etc.</p>

<p>So the sentence should have read <em>"If you measure a large number of things <s>about a small number of people</s>, you are almost guaranteed to get a “statistically significant” result."</em></p>

<p>As correctly calculated in the article, even if chocolate does exactly nothing there was a 60% chance (assuming independence, etc.) of getting a significant result.</p>

<p>They actually got three significant results, which is quite surprising (p=0.06 under the -- probably unrealistic -- assumption of independence).</p>

<p><strong>Does the sample size ever affect the false-positive rate?</strong></p>

<p>Actually sometimes it does, though it only really makes a difference if the sample size is <em>really</em> small.</p>

<p>I said that (assuming that the null hypothesis is true) the p-value should be uniformly distributed. But the uniform distribution is continuous, while a lot of data is discrete with only finitely many possible outcomes.</p>

<p>If I toss a coin a few times to check whether it is biased, there are only a few possible outcomes and hence a few possible p-values, so the distribution of the potential p-values is a very bad approximation to the uniform distribution. If I flip it few enough times, it might be impossible to get a significant result.</p>

<p><a href="http://lesswrong.com/lw/1sk/case_study_abuse_of_frequentist_statistics/" rel="nofollow">Here</a> is an example of a case where that actually happened.</p>

<p>So you would have something like <em>"If you measure certain types of things about a sufficiently small number of people, you are never going to get a “statistically significant” result, no matter how many things you try."</em></p>

<p><strong>Does this mean that you shouldn't worry about the sample size if a result is positive?</strong></p>

<p>No. Some positive results are false positives and some are true positives. As discussed above, it's usually safe to assume that the false-positive rate is fixed (generally at 5%). But a smaller sample size always makes true positives less likely (having a smaller sample size means that the test has lower <a href="http://en.wikipedia.org/wiki/Statistical_power" rel="nofollow">power</a>). And if you have the same number of false positives but fewer true positives, a randomly chosen positive result is more likely to be false.</p>

CC BY-SA 3.0
2015-06-04T22:38:42.203Z
I'm confused by your two last paragraphs. Is it correct that *As sample size decreases, the probability of positives decreases, and the probability of true positives decreases more than the probability of false positives*? If so, how does that match with "*While a small sample can't make false positives more likely, it can make them less likely.*"
FooBar
CC BY-SA 3.0
2015-06-05T06:35:50.090Z
For any vaguely reasonable sample size, it's safe to assume that the false-positive rate is fixed at 5%. But however big your sample, adding more people increases the true-positive rate. I don't have much time now, so I can only make very quick edits, but I'll try to improve the wording properly when I get a chance.
Raoul
CC BY-SA 3.0
2015-06-04T21:39:38.743Z
2015-06-06T08:40:51.600Z
2015-06-06T08:40:51.600Z
Raoul
<p>There is one thing perhaps worth adding to the excellent answers above, which is there is essentially a meta-numbers game going on as well. Lets say that 20 scientists all do the same set of experiments looking for something possibly weakly correlated like "does chocolate cause heart attacks", and will accept the p value &lt; 0.05 significant which frankly they shouldn't. The cumulative probability is that one scientist will get a significant finding, which is the one experiment that will get published, since negative results rarely get accepted. There is then a 100% chance that that finding will get picked up by the Bild Zeitungs of this world and mis-reported. </p>

<p>Unfortunately, because we don't report the absence of findings, we're essentially engaged in a planet wide exercise in reporting all the experiments that get lucky - in the wrong sense of the word.</p>

<p>For subjects with a strong theoretical basis, good experimental design provides some protection against this - for subjects which are pre-dominantly having to work with observational data, and try to work out the theory - like economics - it's a major issue.</p>

<p>Added: For an extensive - and very well written - discussion on the entire issue, see the paper that started the recent debate:</p>

<p><a href="http://rsos.royalsocietypublishing.org/content/1/3/140216" rel="nofollow">An Investigation of the false discovery rate and the misinterpretation of p-values David Colquhoun</a></p>

CC BY-SA 3.0
2015-06-01T16:25:14.903Z
This is a very good remark. The problem plagues findings and meta findings based on earlier papers as well. However I think it is perhaps out of sync with this particular question, as the thought experiment seems to about an individual hack scientist measuring his chances?
Giskard
CC BY-SA 3.0
2015-06-01T16:34:42.197Z
I don't disagree, however since Foobar was asking in the context of a paper he was looking at, I thought it wouldn't hurt to throw in the worst case analysis.
Lumi
CC BY-SA 3.0
2015-06-01T13:40:42.957Z
2015-07-02T13:38:45.260Z
2015-07-02T13:38:45.260Z
Lumi
<p>P-value hacking is the "art" of looking at different outcomes and specifications until you get a "false positive", i.e. a p value under, say, 0.05, which only noise and not true under the data generating process.</p>

<p>Say I have a treated group with size $N$ and a control group with size $M$,  $K$ outcome variables, and am targeting a p-value of $p$: How can I compute the ex-ante probability of getting at least one false positive significant result significant under $p$?</p>

<p>You can assume that the $K$ characteristics are independently and normal distributed, and if it simplifies a lot, that $M=N$. </p>

CC BY-SA 3.0
2015-05-29T18:33:19.303Z
Full disclosure: I am impressed by a fairly interesting result where $M+N = 50$. I'd like to get a rough approximation of how likely their interesting result stems from too many variables of interest.
FooBar
CC BY-SA 3.0
2015-05-29T19:30:22.497Z
What exactly is your null hypothesis? That the average of a given characteristic is the same for both groups? (And this is repeated for all $K$ variables.)
I am not sure but I think you would also have to say something about the type of the underlying probability distribution.
Giskard
CC BY-SA 3.0
2015-05-29T20:38:05.173Z
@cc7768 As far as I understand that article after skimming it, all the results were fake. There is nothing about fake results here, just about rolling the dice often enough. See also http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800
FooBar
CC BY-SA 3.0
2015-05-29T20:41:24.850Z
Foobar, yeah that's why I said possibly relevant haha -- It isn't quite directly related, but your question reminded me of it. Your article seems a little more related :)

@AndréPeseur, I think there is going to be some overlap in topics between our website and cross-validated. I'm of the opinion that econometrics should be on-topic here -- Not a SE pro or anything though. Maybe start a meta post to discuss it further if you disagree.
cc7768
CC BY-SA 3.0
2015-05-29T21:53:51.733Z
@cc7768 don't take him too serious. Andre apparently comes from a statistics background and likes to troll any question/answer of mine that somehow touches statistics by down voting and claiming its NaN/offtopic.
FooBar
CC BY-SA 3.0
2015-05-30T07:09:22.660Z
Another interesting article that illustrates exactly your point: http://opim.wharton.upenn.edu/DPlab/papers/publishedPapers/Simmons_2011_False-Positive%20Psychology.pdf
Oliv
CC BY-SA 3.0
2015-05-30T18:04:43.573Z
If the null hypotheses to be checked are "no difference in means" per characteristic, as was stated in a comment, then the $K$ variables are not "treatment variables", they are just characteristics. There is no treatment here. Do I understand correctly?
Alecos Papadopoulos
CC BY-SA 3.0
2015-05-30T18:13:16.953Z
What I mean is that the treatment variable itself does not enter the picture. It exerted its influence (if any) in the past, and now we are looking at the results.
Alecos Papadopoulos
<econometrics><statistics>
P-value hacking
FooBar
# P-value hacking

(Asked by: username_0 on 2015-05-29)

P-value hacking is the "art" of looking at different outcomes and specifications until you get a "false positive", i.e. a p value under, say, 0.05, which only noise and not true under the data generating process.

Say I have a treated group with size $N$ and a control group with size $M$, $K$ outcome variables, and am targeting a p-value of $p$: How can I compute the ex-ante probability of getting at least one false positive significant result significant under $p$?

You can assume that the $K$ characteristics are independently and normal distributed, and if it simplifies a lot, that $M=N$.

**Tags:** econometrics, statistics

> **Comments:**
> **username_0**: Full disclosure: I am impressed by a fairly interesting result where $M+N = 50$. I'd like to get a rough approximation of how likely their interesting result stems from too many variables of interest.  
> **username_1**: What exactly is your null hypothesis? That the average of a given characteristic is the same for both groups? (And this is repeated for all $K$ variables.) I am not sure but I think you would also have to say something about the type of the underlying probability distribution.  
> **username_0**: @username_2 As far as I understand that article after skimming it, all the results were fake. There is nothing about fake results here, just about rolling the dice often enough. See also http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800  
> **username_2**: username_0, yeah that's why I said possibly relevant haha -- It isn't quite directly related, but your question reminded me of it. Your article seems a little more related :)  @AndréPeseur, I think there is going to be some overlap in topics between our website and cross-validated. I'm of the opinion that econometrics should be on-topic here -- Not a SE pro or anything though. Maybe start a meta post to discuss it further if you disagree.  
> **username_0**: @username_2 don't take him too serious. Andre apparently comes from a statistics background and likes to troll any question/answer of mine that somehow touches statistics by down voting and claiming its NaN/offtopic.  
> **username_3**: Another interesting article that illustrates exactly your point: http://opim.wharton.upenn.edu/DPlab/papers/publishedPapers/Simmons_2011_False-Positive%20Psychology.pdf  
> **username_4**: If the null hypotheses to be checked are "no difference in means" per characteristic, as was stated in a comment, then the $K$ variables are not "treatment variables", they are just characteristics. There is no treatment here. Do I understand correctly?  
> **username_4**: What I mean is that the treatment variable itself does not enter the picture. It exerted its influence (if any) in the past, and now we are looking at the results.  

---

## 4 answers

### Answer by username_4 on 2015-05-30. Score: 11

Under the assumption of i.i.d. Normal characteristics, the situation described is taken care by separate [Welch's t-tests](https://en.wikipedia.org/wiki/Welch%27s_t_test) that account for possibly different sample sizes and different variances. Denote the statistics of these tests $t\_j, j=1,...,K$.
The p-value associated with each is

$$p\_j = \Pr\big(|t\_j|\geq t(\alpha)\mid H\_0\big) $$

where $H\_0$ is the hypothesis that the populations means between treated and controlled group are equal, and $t$ depends on the significance level $1-\alpha$.

We can write the probability in terms of the corresponding cumulative distribution function,

$$\Pr\big(|t\_j|\geq t(\alpha)\mid H\_0\big) = 1 - F(|t\_j|)$$

Therefore

$$p\_j = 1 - F(|t\_j|) \implies 1-p\_j = F(|t\_j|)$$

If we contemplate the situation a priori, before even looking at the data, then the p-values lie in the future and can be modelled as random variables. Viewed as a random variable, the [probability integral transform](https://en.wikipedia.org/wiki/Probability_integral_transform) tells us that $1-p\_j$ follows a $U(0,1)$ Uniform distribution, and by the properties of this distribution so does $p\_j$.

Collecting all $p\_j$, we have a sample of size $K$ of independent $U(0,1)$ uniforms. The probability that at least one of them is smaller than a specific value, say $p^\*$, is equal to the probability that the minimum of them is lower than this threshold. This can be understood as follows:

$$\Pr\Big (\text {At least one $p\_j \leq p^\*$} \Big) = \Pr\Big (\text {Not all $p\_j > p^\*$} \Big) $$

$$ = 1-\Pr\Big (\text {All $p\_j > p^\*$} \Big) = 1- \prod\_{j=1}^K \Pr\Big ( p\_j > p^\* \Big)$$

due to independence, and so, since they are identically distributed,

$$\Pr\Big (\text {At least one $p\_j \leq p^\*$} \Big) = 1- \left [1-\Pr\Big ( p \leq p^\* \Big)\right]^K = 1 - \left [1-F\_U \big(p^\* \big)\right]^K$$

But this is the cumulative distribution function of the minimum of $K$ i.i.d random variables.

Denote this minimum $p\_{(1)}$.

The CDF of the minimum of $K$ independent $U(0,1)$ variables is

$$F\_{p\_{(1)}}(p\_{(1)}) = 1 - \big [1-p\_{(1)}\big]^K$$

We want the probability

$$ \Pr(p\_{(1)} \leq p^\*) = 1- \big [1-p^\*\big]^K$$

Indicative values:

![enter image description here](https://i.sstatic.net/BAnIC.png)

> **Comments:**
> **username_0**: "*Welch's t-tests that account for possibly different sample sizes and different variances*". I had only time to skim the reply so far, but I can't find  where the sample sizes $M$, $N$ enter the picture. How does the last table vary with the sample size?  
> **username_4**: It doesn't. As random variables, the p-values are Uniforms (0,1) whatever the other aspects of the situation. The only thing that matters is the size of $K$.  
> **username_0**: That's strange. [This link](http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800) claims something else: "*If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people.*". What are the underlying assumptions here that lead to there different result?  
> **username_4**: The crucial part of the statement is "a large number of things", i.e. a large $K$. Go back to the link and read just below the sub-chapter "The Hook". You will see the same formula as in my answer, and a percentage consistent with my table.  
> **username_0**: I see. So his formulation "large number of things about a small number of people" implies the wrong thing, it should be without "small number of people"  
> **username_4**: Yes. I would write it as "If you measure a large number of things (about) _even for_ a small number of people, you are almost guaranteed to get a “statistically significant” result." The size of the sample is of course very important, but in relation to other issues.  

---

### Answer by username_5 on 2015-05-31. Score: 6

I agree with @username_4 we want something like:
$$ \Pr(p\_{(1)} \leq p^\*) = 1- \big [1-p^\*\big]^K$$
But I don't see how $n$ and $M$ couldn't enter into the proper test statistic.
For example, if the underlying data is normally distributed i.i.d. data then $N$ and $M$ do matter.

Consider that noise mean $\mu$ and variance $\sigma$, which, by assumption is the same for the control and "treated" group. The mean of the treated group with size N will be distributed $N(\mu, \sigma^2 / n)$ and $N(\mu, \sigma^2 / M)$ for the control. So the difference in means will be distributed $$N(0, \sigma^2 / n + \sigma^2 / m)$$

But you won't know $\sigma$ or $\mu$, so we'll have to estimate it with $X\_1$, $X\_2$, and $s\_{X\_1X\_2}$, and use a t-test. This setup gives a t-statistic like this:
$$t = \frac{\bar {X}\_1 - \bar{X}\_2}{s\_{X\_1 X\_2} \cdot \sqrt{\frac{1}{n}+\frac{1}{m}}}$$
where
$$s\_{X\_1X\_2} = \sqrt{\frac{(n-1)s\_{X\_1}^2+(m-1)s\_{X\_2}^2}{n+m}}.$$
[SRC: Student's t-test on Wikipedia](https://en.wikipedia.org/wiki/Student%27s_t-test)

The unpaired sample t-test for this difference in means has degrees of freedom $N-M-2$. Therefore the rejection region should depend on both n and m, both in what critical value of the test to use through the degrees of freedom of that test and the test statistic calculation itself.

> **Comments:**
> **username_4**: Certainly the sample sizes enter the test-statistic, but this does not affect the specific probability the OP asks about (i.e. the a priori probability that we would obtain at least one p-value lower than a given threshold).  
> **username_5**: A properly specified test would incorporate the multiple comparisons just as it would sample size and degrees of freedom - - right? So is this a question about using the wrong test statistic along only the multiple comparison dimension but properly specified on the individual comparison dimension? Because the presence of m and n in the test statistic is exactly because a small sample size is more likely to have a large difference under the null hypothesis.  
> **username_4**: The crucial point here is that a priori, a) a p-value has a marginal U(0,1) distribution, _irrespective of anything else_ (sample size or whatever). This is a general result holding in any kind of situation. This should be intuitive: what would happen if the p-value had a priori a non-uniform distribution? CONTD  
> **username_4**: CONTD  It would mean that some of its values would be more likely than others, again _irrespective of the specifics of the situation_. But this would invalidate  the whole testing procedure, _exactly because_ it would imply that "it doesn't matter what you test, what  is the sample size etc - _this_ value for the p-value is more probable than others".  
> **username_0**: Thank both of you for the extended discussion, I need to learn more about this and I feel this should definitively be part of a graduate studies curriculum.  
> **username_6**: @username_0 It usually is in Science/Engineering. We got taught explicitly that p < 0.05 wasn't acceptable.  
> **username_4**: @username_6 "p < 0.05 isn't acceptable" : not acceptable for what purpose? What were you taught that it would be "acceptable"?  
> **username_6**: @username_4 Essentially not acceptable to justify publishing without  a strong theoretical basis as well. In fact, he wasn't particularly comfortable even with p < 0.01 just on its own, unless it was a very well designed experiment. This was a very good statistics lecturer who was already quite horrified at the way easy to use computer based statistical packages were essentially enabling p-hacking type approaches.  The run all the statistical tests until you get the magic number approach trap that students quite easily fall into.  
> **username_4**: @username_6 That is a wise approach. My approach to p-values is that they are just one more piece of evidence, not the critical or the leading one.  

---

### Answer by username_7 on 2015-06-04. Score: 3

The other answers are good, but I thought that another answer with a slighlty different focus might be a good complement.

**Does the sample size usually affect the false-positive rate?**

Judging from the comments, I think that the question has been prompted by [this article](http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800), which includes a couple of mistakes (or at least miswordings).

Firstly (and most worryingly in general) it defines p-values incorrectly, but more relevantly it includes the sentence *"If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result."*

The p-value is the probability, assuming that the null hypothesis is true, of observing a result as least as extreme as the one that was actually observed. As pointed out in the other answers, this means that it should be uniformly distributed between 0 and 1 regardless of sample size, underlying distributions, etc.

So the sentence should have read *"If you measure a large number of things ~~about a small number of people~~, you are almost guaranteed to get a “statistically significant” result."*

As correctly calculated in the article, even if chocolate does exactly nothing there was a 60% chance (assuming independence, etc.) of getting a significant result.

They actually got three significant results, which is quite surprising (p=0.06 under the -- probably unrealistic -- assumption of independence).

**Does the sample size ever affect the false-positive rate?**

Actually sometimes it does, though it only really makes a difference if the sample size is *really* small.

I said that (assuming that the null hypothesis is true) the p-value should be uniformly distributed. But the uniform distribution is continuous, while a lot of data is discrete with only finitely many possible outcomes.

If I toss a coin a few times to check whether it is biased, there are only a few possible outcomes and hence a few possible p-values, so the distribution of the potential p-values is a very bad approximation to the uniform distribution. If I flip it few enough times, it might be impossible to get a significant result.

[Here](http://lesswrong.com/lw/1sk/case_study_abuse_of_frequentist_statistics/) is an example of a case where that actually happened.

So you would have something like *"If you measure certain types of things about a sufficiently small number of people, you are never going to get a “statistically significant” result, no matter how many things you try."*

**Does this mean that you shouldn't worry about the sample size if a result is positive?**

No. Some positive results are false positives and some are true positives. As discussed above, it's usually safe to assume that the false-positive rate is fixed (generally at 5%). But a smaller sample size always makes true positives less likely (having a smaller sample size means that the test has lower [power](http://en.wikipedia.org/wiki/Statistical_power)). And if you have the same number of false positives but fewer true positives, a randomly chosen positive result is more likely to be false.

> **Comments:**
> **username_0**: I'm confused by your two last paragraphs. Is it correct that *As sample size decreases, the probability of positives decreases, and the probability of true positives decreases more than the probability of false positives*? If so, how does that match with "*While a small sample can't make false positives more likely, it can make them less likely.*"  
> **username_7**: For any vaguely reasonable sample size, it's safe to assume that the false-positive rate is fixed at 5%. But however big your sample, adding more people increases the true-positive rate. I don't have much time now, so I can only make very quick edits, but I'll try to improve the wording properly when I get a chance.  

---

### Answer by username_6 on 2015-06-01. Score: 2

There is one thing perhaps worth adding to the excellent answers above, which is there is essentially a meta-numbers game going on as well. Lets say that 20 scientists all do the same set of experiments looking for something possibly weakly correlated like "does chocolate cause heart attacks", and will accept the p value < 0.05 significant which frankly they shouldn't. The cumulative probability is that one scientist will get a significant finding, which is the one experiment that will get published, since negative results rarely get accepted. There is then a 100% chance that that finding will get picked up by the Bild Zeitungs of this world and mis-reported.

Unfortunately, because we don't report the absence of findings, we're essentially engaged in a planet wide exercise in reporting all the experiments that get lucky - in the wrong sense of the word.

For subjects with a strong theoretical basis, good experimental design provides some protection against this - for subjects which are pre-dominantly having to work with observational data, and try to work out the theory - like economics - it's a major issue.

Added: For an extensive - and very well written - discussion on the entire issue, see the paper that started the recent debate:

[An Investigation of the false discovery rate and the misinterpretation of p-values David Colquhoun](http://rsos.royalsocietypublishing.org/content/1/3/140216)

> **Comments:**
> **username_1**: This is a very good remark. The problem plagues findings and meta findings based on earlier papers as well. However I think it is perhaps out of sync with this particular question, as the thought experiment seems to about an individual hack scientist measuring his chances?  
> **username_6**: I don't disagree, however since username_0 was asking in the context of a paper he was looking at, I thought it wouldn't hurt to throw in the worst case analysis.  

---

