Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to
perform new tasks with minimal supervision. However, ICL performance,
especially in smaller LMMs, is inconsistent and does not always improve
monotonically with increasing examples. We hypothesize that this occurs due to
the LMM being overwhelmed by additional information present in the image
embeddings, which is not required for the downstream task. To address this, we
propose a meta-learning approach that provides an alternative for inducing
few-shot capabilities in LMMs, using a fixed set of soft prompts that are
distilled from task-relevant image features and can be adapted at test time
using a few examples. To facilitate this distillation, we introduce an
attention-mapper module that can be easily integrated with the popular LLaVA
v1.5 architecture and is jointly learned with soft prompts, enabling task
adaptation in LMMs under low-data regimes with just a few gradient steps.
Evaluation on the VL-ICL Bench shows that our method consistently outperforms
ICL and related prompt-tuning approaches, even under image perturbations,
improving task induction and reasoning across visual question answering tasks.
6847d8503ec10bdd8ab4dfd7
Akash Gupta
claimed_verified
2025-06-10T08:42:36.181Z
638a5b46002d85749924d5c8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/YA4i44NwKidCTrCANMvMB.png
Akash Gupta
user
aksgupta97
6847d8503ec10bdd8ab4dfd8
Amos Storkey
6847d8503ec10bdd8ab4dfd9
Mirella Lapata
Meta-Adaptive Prompt Distillation for Few-Shot Visual Question Answering
Large Multimodal Models (LMMs) often rely on in-context learning (ICL) to
perform new tasks with minimal supervision. However, ICL performance,
especially in smaller LMMs, is inconsistent and does not always improve
monotonically with increasing examples. We hypothesize that this occurs due to
the LMM being overwhelmed by additional information present in the image
embeddings, which is not required for the downstream task. To address this, we
propose a meta-learning approach that provides an alternative for inducing
few-shot capabilities in LMMs, using a fixed set of soft prompts that are
distilled from task-relevant image features and can be adapted at test time
using a few examples. To facilitate this distillation, we introduce an
attention-mapper module that can be easily integrated with the popular LLaVA
v1.5 architecture and is jointly learned with soft prompts, enabling task
adaptation in LMMs under low-data regimes with just a few gradient steps.
Evaluation on the VL-ICL Bench shows that our method consistently outperforms
ICL and related prompt-tuning approaches, even under image perturbations,
improving task induction and reasoning across visual question answering tasks.
A meta-learning approach with soft prompts and an attention-mapper module improves few-shot capabilities in LMMs, outperforming ICL and related methods in visual question answering tasks.
in-context learning meta-learning soft prompts attention-mapper module LLaVA few-shot capabilities task adaptation low-data regimes gradient steps visual question answering