S3: A Simple Strong Sample-effective Multimodal Dialog System
<|reference_start|>S3: A Simple Strong Sample-effective Multimodal Dialog System: In this work, we present a conceptually simple yet powerful baseline for the multimodal dialog task, an S3 model, that achieves near state-of-the-art results on two compelling leaderboards: MMMU and AI Journey Contest 2023. The system is based on a pre-trained large language model, pre-trained modality encoders for image and audio, and a trainable modality projector. The proposed effective data mixture for training such an architecture demonstrates that a multimodal model based on a strong language model and trained on a small amount of multimodal data can perform efficiently in the task of multimodal dialog.<|reference_end|>
@article{rykov2024s3:,
    title={S3: A Simple Strong Sample-effective Multimodal Dialog System},
    author={Elisei Rykov, Egor Malkershin, and Alexander Panchenko},
    journal={arXiv preprint arXiv:2406.18305},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.18305},
    primaryClass={cs.CL cs.AI}
}