Multitask and Multimodal Neural Tuning for Large Models
<|reference_start|>Multitask and Multimodal Neural Tuning for Large Models: In recent years, large-scale multimodal models have demonstrated impressive capabilities across various domains. However, enabling these models to effectively perform multiple multimodal tasks simultaneously remains a significant challenge. To address this, we introduce a novel tuning method called neural tuning, designed to handle diverse multimodal tasks concurrently, including reasoning segmentation, referring segmentation, image captioning, and text-to-image generation. Neural tuning emulates sparse distributed representation in human brain, where only specific subsets of neurons are activated for each task. Additionally, we present a new benchmark, MMUD, where each sample is annotated with multiple task labels. By applying neural tuning to pretrained large models on the MMUD benchmark, we achieve simultaneous task handling in a streamlined and efficient manner. All models, code, and datasets will be publicly available after publication, facilitating further research and development in this field.<|reference_end|>
@article{sun2024multitask,
    title={Multitask and Multimodal Neural Tuning for Large Models},
    author={Hao Sun, Yu Song, Jihong Hu, Yen-Wei Chen, and Lanfen Lin},
    journal={arXiv preprint arXiv:2408.03001},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.03001},
    primaryClass={cs.CV cs.MM}
}