Calculating a P value This question is focused on seemingly difficult probability calculations in experiments for calculating p-values.
In the simple example of finding out if a 6 sided die is a fair or loaded die, one can easily calculate the probability of getting 10 1s in a row after rolling the die 10 times and determine if that value meets the requisite threshold for statistical significance.
However, how are problems approached that may have more difficult probability calculations? A 6 sided die is exceedingly simple, but other experiments may have more difficult probability calculations while assuming the null hypothesis.
For instance, does some new drug reduce the size of a cancer tumor. Would one have to gather a huge database of patients with cancer tumors and calculate over a 30 day period what the typical size change of such tumors is? But when designing the experiment to get the typical (non drug induced) change in tumors size, how does one know when that experiment gets to statistical significance? Since in that case we are AGAIN estimating a statistic from a sample of cancer patients.
It seems there is ample opportunity in a scientific process like the above to accumulate errors and thus distort the end result of whether such a drug works.
How is this issue handled in the real world.

For complicated settings it is important to validate your modelling assumptions.  If your distributional assumptions appear correct, then you have confidence that although you may have not selected the true distribution, you have adequately approximated the true distribution, and hence your p-values are still relevant.
