Optimizing importance weighting in the presence of sub-population shifts
Floris Holstege Bram Wouters Noud Van Giersbergen Cees Diks
A distribution shift between the training and test data can severely harm performance of machine learning models. Importance weighting addresses this issue by assigning different weights to data points during training. We argue that existing heuristics for determining the weights are suboptimal, as they neglect the increase of the variance of the estimated model due to the limited sample size of the training data. We interpret the optimal weights in terms of a bias-variance trade-off,  and  propose a bi-level optimization procedure in which the weights and model parameters are optimized simultaneously. We apply this framework to existing importance weighting techniques for last-layer retraining of deep neural networks in the presence of sub-population shifts and show empirically that optimizing weights significantly improves generalization performance.
Importance weighting, Distribution shift, Group Robustness, Bi-level optimization
We show that existing importance weighting methods for sub-population shift can be improved using a bi-level optimization procedure over the weights