P values are not as reliable as many scientists assume
I was recently trying to explain Bayesian logic to a friend, and came up with the following analogy. I would be interested to hear feedback on it.<p>---<p>Imagine everyone in the USA gets sudden amnesia. We want to find out who the President is, but no one can remember.<p>A scientist comes up with a test to determine if someone is the President.<p>If they <i>are</i> the President, there is a 100% chance the test will say they are the President and a 0% chance the test will say they are not the President.<p>If they are <i>not</i> the President, there is a 99.999% chance the test will say that are not the President, and a 0.001% chance the test will falsely say they are the President.<p>Giving the test to the person sitting in the big chair in the Oval Office is useful, because it's already quite likely this person is the President. If the test is positive for Presidency, it's extremely likely that person is the president.<p>Giving the test to the 10 people nearest the oval office is useful, because it's fairly likely the President is one of these people. A positive result will indicate strongly that that person is the President, and if no-one in that group is actually the President, there's a 99.99% chance the test will say so.<p>Giving the test to the 1000 people in the White House is pretty useful, because it's pretty likely the President is in the White House, and if none of these people are the president, there's still a 99% chance the test will be correct. A positive result for any one person will indicate quite strongly that that person is the President.<p>But giving the test to everyone in America is not very useful at all, because it's very unlikely that any particular person is the President, and we can expect the test will give a positive result for around 3200 people. For any particular person in this group, it's much more likely they're not the President than they are.<p>---<p>Is this a broadly correct, if non-rigorous, analogy? I realize most HNers will be much more familiar with this stuff than I am, I'm interested chiefly in whether or not I misled my friend.