BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
<|reference_start|>BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models: Large Language Models (LLMs) often suffer from overconfidence during inference, particularly when adapted to downstream domain-specific tasks with limited data. Previous work addresses this issue by employing approximate Bayesian estimation after the LLMs are trained, enabling them to quantify uncertainty. However, such post-training approaches' performance is severely limited by the parameters learned during training. In this paper, we go beyond post-training Bayesianization and propose Bayesian Low-Rank Adaptation by Backpropagation (BLoB), an algorithm that continuously and jointly adjusts both the mean and covariance of LLM parameters throughout the whole fine-tuning process. Our empirical results verify the effectiveness of BLoB in terms of generalization and uncertainty estimation, when evaluated on both in-distribution and out-of-distribution data.<|reference_end|>
@article{wang2024blob:,
    title={BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language
  Models},
    author={Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, Hao Wang},
    journal={arXiv preprint arXiv:2406.11675},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11675},
    primaryClass={cs.LG cs.AI cs.CL stat.ML}
}