Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples
<|reference_start|>Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples: Label corruption, where training samples have incorrect labels, can significantly degrade the performance of machine learning models. This corruption often arises from non-expert labeling or adversarial attacks. Acquiring large, perfectly labeled datasets is costly, and retraining large models from scratch when a clean dataset becomes available is computationally expensive. To address this challenge, we propose Post-Training Correction, a new paradigm that adjusts model parameters after initial training to mitigate label noise, eliminating the need for retraining. We introduce Verifix, a novel Singular Value Decomposition (SVD) based algorithm that leverages a small, verified dataset to correct the model weights using a single update. Verifix uses SVD to estimate a Clean Activation Space and then projects the model's weights onto this space to suppress activations corresponding to corrupted data. We demonstrate Verifix's effectiveness on both synthetic and real-world label noise. Experiments on the CIFAR dataset with 25% synthetic corruption show 7.36% generalization improvements on average. Additionally, we observe generalization improvements of up to 2.63% on naturally corrupted datasets like WebVision1.0 and Clothing1M.<|reference_end|>
@article{kodge2024verifix:,
    title={Verifix: Post-Training Correction to Improve Label Noise Robustness with
  Verified Samples},
    author={Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, Kaushik Roy},
    journal={arXiv preprint arXiv:2403.08618},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.08618},
    primaryClass={cs.LG cs.AI stat.ML}
}