Testing if a sample follows a given distribution Suppose I have a single sample statistic $y$ (e.g., a scalar computed from the data collected after conducting an experiment) and would like to "test" if $y$ follows a given probability density function $p(x)$ (this is of course ill-posed - see below).

*

*How can I formally express this as a hypothesis testing problem? Also, is this unique? i.e., are there multiple ways of casting this task (described in words as "test if $y \sim p(x)$") as hypothesis testing problems?

*How do I do this for the special cases of (a) $p(x)= \mathcal{N}(\mu,\sigma^2)$ and (b) $p(x) = \chi^2_k$?


All standard hypothesis testing follows the following steps:

*

*Formulate a null hypothesis. This should rigorously state conditions under which one can calculate probabilities (e.g. "The observations are all drawn from independent normal distributions with $\mu = 0$ and $\sigma = 1$.")

*Choose an $\alpha$

*Choose a statistical variable $X$

*Collect data

*Calculate the statistic $x$ from the data

*Reject the null if $P(X\ge x|H_0)\le\alpha$ (Note that $X$ is a random variable with a probability distribution, while $x$ is a particular value that that variable takes; $P(X\ge x|H_0)$ is the probability, given the null hypothesis, of getting something as or more extreme than what you actually saw.)

So to test whether a sample follows a given distribution, you need some metric for measuring "distance" from a given distribution, and then you use that as your statistic.

How can I formally express this as a hypothesis testing problem?

All you need is some function $f: ((\mathbb R \rightarrow \mathbb R) \times \mathbb R ^n) \rightarrow \mathbb R$. That is, a function that takes an ordered pair as input and gives a real number as output, where the first element of the ordered pair is a PDF and the second element is a sample of $n$ observations. Once you have such a function, you can find what that function gives for your sample, then calculate the probability, given the null hypothesis (the null hypothesis in this case is that the sample came from the proposed distribution) that you would get a sample for which the function would give a result greater than or equal to what you actually got.

Also, is this unique?

No. Every function gives a different test. Two attributes to look for in choosing the function are tractability (how easy is it to calculate $P(X\ge x|H_0)$) and power (how well does the function correspond to what we think of "agreeing" with the distribution). If you think someone is deliberately trying to fake a particular distribution, then you might want a function tailored to what errors you think they might make (for instance, someone faking a normal distribution might focus on getting the mean and sd correct, but ignore the kurtosis).
