How to correct Chi-square's p-value when working with very unbalanced contingency tables? I'm studying the association between a rare disease and smoking. Because the disease is rare, my contingency table is highly unbalanced with way more Non-Diseased than Diseased individuals, independently of their smoker status.
            NonDiseased | Diseased
___________ ____________ __________
Smoker     |    4312    |    16
___________ ____________ __________
Non-Smoker |   21329    |    20
___________ ____________ __________

Is there a way to correct the p-value of a chi-square test done on this table to reflect the fact that there are very few Diseased individuals?

The bottom line is that the proportions of diseased subjects among smokers and non-smokers are 0.0037 and  0.0011, respectively, and they are highly significantly different.
Because counts 16 and 20 are relatively small some statisticians might use the Yates continuity correction, which is conservative (making the chi-sq statistic smaller, hence the P-value larger). With or without this 'correction' your P-value is very small.
Computations in R below:
TBL
      [,1] [,2]
smok  4312   16
nons 21329   20


chisq.test(TBL)

        Pearson's Chi-squared test 
        with Yates' continuity correction

data:  TBL
X-squared = 17.658, df = 1, p-value = 2.644e-05

chisq.test(TBL, cor=F)

        Pearson's Chi-squared test

data:  TBL
X-squared = 19.58, df = 1, p-value = 9.649e-06

The expected counts (all larger than 5) in this chi-squared test are sufficiently large for a good approximation of the null distribution
to $\mathsf{Chisq}(\nu = 1).$
chisq.test(TBL, cor=F)$exp
          [,1]      [,2]
smok  4321.932  6.067999
nons 21319.068 29.932001

