Inna Gerlovina|Mark J. van der Laan|Alun Hubbard
Big Data, Small Sample
University of California, Berkeley|University of California, Berkeley|Berkeley Public Health Division|University of California, Berkeley
multiple comparisons small sample size common characteristics many types big data including produced genomic studies present specific challenges affect reliability inference use multiple testing procedures necessitates calculation small tail probabilities test statistic distribution results based large deviation theory provide formal condition necessary guarantee error rate control given practical sample sizes linking number tests sample size condition however rarely satisfied using methods based edgeworth expansions relying especially work peter hall explore impact departures sampling distributions typical assumptions actual error rates investigation illustrates far actual error rates declared nominal levels suggesting potentially widespread problems error rate control specifically excessive false positives important factor contributes reproducibility crisis also review commonly used methods permutation methods based finite sampling inequalities application multiple testingsmall sample data point edgeworth expansions providing higher order approximations sampling distribution offer promising direction data analysis could improve reliability studies relying large numbers comparisons modest sample sizes