GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via
  Reinforcement Learning
Medical visual question answering aims to support clinical decision-making by
enabling models to answer natural language questions based on medical images.
While recent advances in multi-modal learning have significantly improved
performance, current methods still suffer from limited answer reliability and
poor interpretability, impairing the ability of clinicians and patients to
understand and trust model-generated answers. To address this, this work first
proposes a Thinking with Visual Grounding (ThinkVG) dataset wherein the answer
generation is decomposed into intermediate reasoning steps that explicitly
ground relevant visual regions of the medical image, thereby providing
fine-grained explainability. Furthermore, we introduce a novel verifiable
reward mechanism for reinforcement learning to guide post-training, improving
the alignment between the model's reasoning process and its final answer.
Remarkably, our method achieves comparable performance using only one-eighth of
the training data, demonstrating the efficiency and effectiveness of the
proposal. The dataset is available at
https://huggingface.co/datasets/BoKelvin/GEMeX-ThinkVG.
685a9d2c0e4ad7e219758756
Bo Liu
claimed_verified
2025-06-24T14:43:47.534Z
61e6dd8a82b19b93e1a51fa6
/avatars/babbee52793a35dd5754d000946dd1ee.svg
Kelvin Liu
user
BoKelvin
685a9d2c0e4ad7e219758757
Xiangyu Zhao
685a9d2c0e4ad7e219758758
Along He
685a9d2c0e4ad7e219758759
Yidi Chen
685a9d2c0e4ad7e21975875a
Huazhu Fu
685a9d2c0e4ad7e21975875b
Xiao-Ming Wu
GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via
  Reinforcement Learning
Medical visual question answering aims to support clinical decision-making by
enabling models to answer natural language questions based on medical images.
While recent advances in multi-modal learning have significantly improved
performance, current methods still suffer from limited answer reliability and
poor interpretability, impairing the ability of clinicians and patients to
understand and trust model-generated answers. To address this, this work first
proposes a Thinking with Visual Grounding (ThinkVG) dataset wherein the answer
generation is decomposed into intermediate reasoning steps that explicitly
ground relevant visual regions of the medical image, thereby providing
fine-grained explainability. Furthermore, we introduce a novel verifiable
reward mechanism for reinforcement learning to guide post-training, improving
the alignment between the model's reasoning process and its final answer.
Remarkably, our method achieves comparable performance using only one-eighth of
the training data, demonstrating the efficiency and effectiveness of the
proposal. The dataset is available at
https://huggingface.co/datasets/BoKelvin/GEMeX-ThinkVG.
A novel dataset and verifiable reward mechanism enhance the explainability and efficiency of medical visual question answering models.
Thinking with Visual Grounding verifiable reward mechanism reinforcement learning intermediate reasoning steps explainability medical visual question answering