CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning
<|reference_start|>CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning: Visual instruction tuning is a key training stage of large multimodal models (LMMs). Nevertheless, the common practice of indiscriminately mixing instruction-following data from various tasks may result in suboptimal overall performance due to different instruction formats and knowledge domains across tasks. To mitigate this issue, we propose a novel Comprehensive Task Balancing (CoTBal) algorithm for multi-task visual instruction tuning of LMMs. To our knowledge, this is the first work that explores multi-task optimization in visual instruction tuning. Specifically, we consider two key dimensions for task balancing: (1) Inter-Task Contribution, the phenomenon where learning one task potentially enhances the performance in other tasks, attributable to the overlapping knowledge domains, and (2) Intra-Task Difficulty, which refers to the learning difficulty within a single task. By quantifying these two dimensions with performance-based metrics, task balancing is thus enabled by assigning more weights to tasks that offer substantial contributions to others, receive minimal contributions from others, and also have great intra-task difficulties. Experiments show that our CoTBal leads to superior overall performance in multi-task visual instruction tuning.<|reference_end|>
@article{dai2024cotbal:,
    title={CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction
  Tuning},
    author={Yanqi Dai, Dong Jing, Nanyi Fei, Zhiwu Lu},
    journal={arXiv preprint arXiv:2403.04343},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04343},
    primaryClass={cs.AI}
}