Multivariate sampling bias correction for data analysis I have a dataset of properties for 20,000 samples and 200 of these samples have a categorical label (A, B or C); and I'm trying to assess the relationships between the properties and the labels. I am doing this with Chi2 and Kendal's Tau tests for the discrete and continuous properties respectively. However, there is a significant sampling bias i.e. the 200 samples which have been assigned labels are not representative of the underlying population as assigning a label requires the sample to be tested and there are various factors influencing the decision to test a sample.
I have two questions:

*

*Does this sampling bias invalidate the Chi2 and Kendal Tau tests of feature importance?

*If so, are there methods for correcting this sampling bias? The underlying distributions are known for all the properties as the unlabelled + labelled data is the entire population, and I feel like there should be some way of using this knowledge to correct the bias in the labelled data but I can't seem to find any appropriate methods.

Edit: The method described in "Huang, Jiayuan, et al. "Correcting sample selection bias by unlabeled data." Advances in neural information processing systems 19 (2006)." appears to be one way of correcting this bias; though ideally there would be a robust implementation of this somewhere, which I am struggling to find.

According to [1], $P(s=1|z)$ --- the probability that an instance is included in the sample given its feature values --- can be used to weight the biased data.
Fortunately in my case, the factors which influence the decision to test a sample are discrete variables. This means that, again following the suggestion in [1], there is a simple method to calculate $P(s=1|z)$.
Let $X$ be the input/predictor space and $Y$ be the label space and $z = (x,y)$ where $x\in X$ and $y\in Y$. If we assume the sampling probability is independent of the labels we can calculate $P(s=1|z)$ as $m_{x}/n_{x}$ where $m_{x}$ is the number of times $x$ appears in the biased sample and $n_{x}$ is the number of times $x$ appears in the underlying population/unbiased sample (in my case Labelled + Unlabelled Data).
[1] Cortes, Corinna, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. ‘Sample Selection Bias Correction Theory’. In Algorithmic Learning Theory, edited by Yoav Freund, László Györfi, György Turán, and Thomas Zeugmann, 5254:38–53. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008. https://doi.org/10.1007/978-3-540-87987-9_8.
Something like the following in Python will add a weight column to a dataframe
import pandas as pd
from typing import List


def append_weight_column(biased_data: pd.DataFrame, biasing_vars: List[str], all_data: pd.DataFrame) -> pd.DataFrame:
    # Get different combinations of biasing variables and how often they occur in biased data and all data
    biased_data_combinations = biased_data.groupby(biasing_vars, as_index=False, dropna=False).size()
    all_data_combinations = all_data.groupby(biasing_vars, as_index=False, dropna=False).size()

    # Merge on these biasing variables to add n and m columns
    m = biased_data.merge(all_data_combinations, on=biasing_vars, how='left')
    m.rename(columns={'size': 'n'}, inplace=True)
    out = m.merge(biased_data_combinations, on=biasing_vars, how='left')
    out.rename(columns={'size': 'm'}, inplace=True)

    out['weight'] = out['n'] / out['m']

    out.drop(columns=['m', 'n'], inplace=True)
    return out

