Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge
<|reference_start|>Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge: In this paper, we present our solution for the WSDM2023 Toloka Visual Question Answering Challenge. Inspired by the application of multimodal pre-trained models to various downstream tasks(e.g., visual question answering, visual grounding, and cross-modal retrieval), we approached this competition as a visual grounding task, where the input is an image and a question, guiding the model to answer the question and display the answer as a bounding box on the image. We designed a three-stage solution for this task. Specifically, we used the visual-language pre-trained model OFA as the foundation. In the first stage, we constructed a large-scale synthetic dataset similar to the competition dataset and coarse-tuned the model to learn generalized semantic information. In the second stage, we treated the competition task as a visual grounding task, loaded the weights from the previous stage, and continued to fine-tune the model on the competition dataset, transferring the semantic information learned in the first stage to the competition task. Finally, we designed a bounding box matching and replacing post-processing strategy to correct the model's prediction results. Our team achieved a score of 76.342 on the final leaderboard, ranking second.<|reference_end|>
@article{wu2024second,
    title={Second Place Solution of WSDM2023 Toloka Visual Question Answering
  Challenge},
    author={Xiangyu Wu, Zhouyang Chi, Yang Yang, Jianfeng Lu},
    journal={arXiv preprint arXiv:2407.04255},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.04255},
    primaryClass={cs.CV}
}