VideoICL: Confidence-based Iterative In-context Learning for
  Out-of-Distribution Video Understanding
Recent advancements in video large multimodal models (LMMs) have
significantly improved their video understanding and reasoning capabilities.
However, their performance drops on out-of-distribution (OOD) tasks that are
underrepresented in training data. Traditional methods like fine-tuning on OOD
datasets are impractical due to high computational costs. While In-context
learning (ICL) with demonstration examples has shown promising generalization
performance in language tasks and image-language tasks without fine-tuning,
applying ICL to video-language tasks faces challenges due to the limited
context length in Video LMMs, as videos require longer token lengths. To
address these issues, we propose VideoICL, a novel video in-context learning
framework for OOD tasks that introduces a similarity-based relevant example
selection strategy and a confidence-based iterative inference approach. This
allows to select the most relevant examples and rank them based on similarity,
to be used for inference. If the generated response has low confidence, our
framework selects new examples and performs inference again, iteratively
refining the results until a high-confidence response is obtained. This
approach improves OOD video understanding performance by extending effective
context length without incurring high costs. The experimental results on
multiple benchmarks demonstrate significant performance gains, especially in
domain-specific scenarios, laying the groundwork for broader video
comprehension applications. Code will be released at
https://github.com/KangsanKim07/VideoICL
674fc0a016234f46d24f48e1
Kangsan Kim
claimed_verified
2024-12-05T09:28:23.078Z
66ed7737f2f27a5dfd81ef09
/avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
Kangsan Kim
user
KangsanKim71
674fc0a016234f46d24f48e2
Geon Park
claimed_verified
2024-12-06T09:09:48.617Z
646cae3093badbc8c2e891c7
/avatars/4aae2aca70ea9dc58dd6f9f9b2be15e1.svg
Geon Park
user
geonp
674fc0a016234f46d24f48e3
Youngwan Lee
674fc0a016234f46d24f48e4
Woongyeong Yeo
claimed_verified
2024-12-04T09:45:38.836Z
66d30f5fad293ffc4b7672bc
/avatars/6f164d813b947940a088820f8fd4dbe8.svg
Woongyeong Yeo
user
wgcyeo
674fc0a016234f46d24f48e5
Sung Ju Hwang
VideoICL: Confidence-based Iterative In-context Learning for
  Out-of-Distribution Video Understanding
Recent advancements in video large multimodal models (LMMs) have
significantly improved their video understanding and reasoning capabilities.
However, their performance drops on out-of-distribution (OOD) tasks that are
underrepresented in training data. Traditional methods like fine-tuning on OOD
datasets are impractical due to high computational costs. While In-context
learning (ICL) with demonstration examples has shown promising generalization
performance in language tasks and image-language tasks without fine-tuning,
applying ICL to video-language tasks faces challenges due to the limited
context length in Video LMMs, as videos require longer token lengths. To
address these issues, we propose VideoICL, a novel video in-context learning
framework for OOD tasks that introduces a similarity-based relevant example
selection strategy and a confidence-based iterative inference approach. This
allows to select the most relevant examples and rank them based on similarity,
to be used for inference. If the generated response has low confidence, our
framework selects new examples and performs inference again, iteratively
refining the results until a high-confidence response is obtained. This
approach improves OOD video understanding performance by extending effective
context length without incurring high costs. The experimental results on
multiple benchmarks demonstrate significant performance gains, especially in
domain-specific scenarios, laying the groundwork for broader video
comprehension applications. Code will be released at
https://github.com/KangsanKim07/VideoICL
VideoICL enhances out-of-distribution video understanding with similarity-based example selection and iterative inference, extending context length without high computational costs.
Video large multimodal models LMMs out-of-distribution (OOD) tasks in-context learning (ICL) video in-context learning similarity-based example selection confidence-based iterative inference domain-specific scenarios video comprehension