Determine if a categorical variable occurs at a frequency greater than chance with a boolean outcome I am looking for the proper statistic to answer whether which categorical items (from a long list) occur at frequency greater than chance with each state of a boolean variable.
More specifically, I have a long list of medications patients have been taken and diseases they have been diagnosed with. I would like a statistical test that answers "which of these drugs are present with each disease at frequency greater than chance" with associated p-values and confidence intervals for each medication and disease-state combination.
There is over 5,000 medications so ideally the statistical approach could include an correction for the high number of comparisons (if appropriate).
Sample Data




patient
medication
has cancer (boolean)
has heart disease(boolean)




patient A
drug A
False
True


patient A
drug B
False
True


patient B
drug A
True
True


patient C
drug C
False
False




Desired output
p-value for Drug C co-occurring greater than chance with has heart disease=True is:
p-value for Drug A co-occurring greater than chance with has cancer=False is:

Let's start with the following (sloppy) hypothesis:

*

*$H_0$: Drug $A$ has the same probability of being prescribed when a patient has disease $C$ as for any other disease.
Let's assume that sufficient data are available and let $X$ be your data matrix consisting of $n$ rows.

*

*Estimate the probability that drug $A$ is prescribed, regardless of the underlying disease. This is given by $\hat{p}_A=\frac{n_A}{n}$, where $n_A$ is the number of rows in $X$ containing drug $A$.

*Estimate the probability that a patient has disease $C$,regardless of the drug prescribed. This is given by $\hat{p}_C=\frac{n_C}{n}$, where $n_C$ is the number of rows in $X$ where disease $C$ is true.

*Count the number of patients that receive $A$ and suffer from $C$ as $n_{C|A}$.

Since your data are Boolean, $n_A$,$p_A$, $n_{C}$ and $p_{C}$ are the parameters of two binomial distributions, and $\hat{p}_A,\hat{p}_{C}$ are their maximum likelihood estimates (assuming that a patient's probability of suffering from disease $C$ is independent from all other patients). Let $q$ be the quantile function of the binomial distribution (python implementation,R implementation)


*Select a confidence level $\alpha$, e.g. $\alpha=0.05$, and compute decision boundaries $b_1=q(0.5\alpha,n,\hat{p}_A\cdot\hat{p}_C)$ and $b_2=q(1-0.5\alpha,n,\hat{p}_A\cdot\hat{p}_C)$
Our proper hypothesis test is the following:


*Reject $H_0: p_{C|A}=p_{C}\cdot p_A$ versus $H_1: p_{C|A}\neq p_{C}\cdot p_A$ if and only if $n_{C|A} \le b_1$ or $n_{C|A} \ge b_2$.

Example: if $500$ out of $1000$ patients receive $A$, and $60$ out of all $1000$ patients suffer from $C$, then we get $\hat{p}_A=0.5$ and $\hat{p}_{C}=0.06$. Now, if $10$ patients receive $A$ and suffer from $C$, we have $n_{C|A}=10$. For $\alpha=0.05$, the decision boundaries are $$b_1=q(0.025,1000,0.5\cdot0.06)=20 \quad b_2=q(0.975,1000,0.5\cdot0.06)=41.$$ So, on a chance level, we would expect that more that $20$ and less that $41$ patients that get $A$ also suffer from $C$. However, we observed $n_{C|A}=10$, so drug $A$ is not as likely to be prescribed when a patient has $C$ as for a random patient - in fact, it is less likely.


*Repeat this hypothesis test for any drug-disease combination you are interested in, assuming that you have sufficient data.


*If you repeat this for many drugs, you should control for the number of hypotheses your are testing. So if you conduct $m$ tests and want total significance level $\alpha$, use $\frac{\alpha}{m}$ in all individual tests instead
