Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning
<|reference_start|>Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning: In this paper, we present our solution for SMART-101 Challenge of CVPR Multi-modal Algorithmic Reasoning Task 2024. Unlike traditional visual questions and answer tasks, this challenge evaluates abstraction, deduction and generalization ability of neural network in solving visuo-linguistic puzzles designed for specially children in the 6-8 age group. Our model is based on two pre-trained models, dedicated to extract features from text and image respectively. To integrate the features from different modalities, we employed a fusion layer with attention mechanism. We explored different text and image pre-trained models, and fine-tune the integrated classifier on the SMART-101 dataset. Experiment results show that under the data splitting style of puzzle split, our proposed integrated classifier achieves superior performance, verifying the effectiveness of multi-modal pre-trained representations.<|reference_end|>
@article{zhang2024integrating,
    title={Integrating Text and Image Pre-training for Multi-modal Algorithmic
  Reasoning},
    author={Zijian Zhang and Wei Liu},
    journal={arXiv preprint arXiv:2406.05318},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05318},
    primaryClass={cs.CV cs.AI}
}