Enhancing Domain Adaptation through Prompt Gradient Alignment
<|reference_start|>Enhancing Domain Adaptation through Prompt Gradient Alignment: Prior Unsupervised Domain Adaptation (UDA) methods often aim to train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, a line of works based on prompt learning leverages the power of large-scale pre-trained vision-language models to learn both domain-invariant and specific features through a set of domain-agnostic and domain-specific learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space to learn such prompts. Differently, we cast UDA as a multiple-objective optimization problem in which each objective is represented by a domain loss. Under this new framework, we propose aligning per-objective gradients to foster consensus between them. Additionally, to prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, we devise a practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently surpasses other prompt-based baselines by a large margin on different UDA benchmarks<|reference_end|>
@article{phan2024enhancing,
    title={Enhancing Domain Adaptation through Prompt Gradient Alignment},
    author={Hoang Phan, Lam Tran, Quyen Tran, Trung Le},
    journal={arXiv preprint arXiv:2406.09353},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09353},
    primaryClass={cs.LG cs.CV}
}