title: VideoICL: Confidence-based Iterative In-context Learning for
    Out-of-Distribution Video Understanding
summary: Recent advancements in video large multimodal models (LMMs) have
  significantly improved their video understanding and reasoning capabilities.
  However, their performance drops on out-of-distribution (OOD) tasks that are
  underrepresented in training data. Traditional methods like fine-tuning on OOD
  datasets are impractical due to high computational costs. While In-context
  learning (ICL) with demonstration examples has shown promising generalization
  performance in language tasks and image-language tasks without fine-tuning,
  applying ICL to video-language tasks faces challenges due to the limited
  context length in Video LMMs, as videos require longer token lengths. To
  address these issues, we propose VideoICL, a novel video in-context learning
  framework for OOD tasks that introduces a similarity-based relevant example
  selection strategy and a confidence-based iterative inference approach. This
  allows to select the most relevant examples and rank them based on similarity,
  to be used for inference. If the generated response has low confidence, our
  framework selects new examples and performs inference again, iteratively
  refining the results until a high-confidence response is obtained. This
  approach improves OOD video understanding performance by extending effective
  context length without incurring high costs. The experimental results on
  multiple benchmarks demonstrate significant performance gains, especially in
  domain-specific scenarios, laying the groundwork for broader video
  comprehension applications. Code will be released at
  https://github.com/KangsanKim07/VideoICL
paper_authors._id: 674fc0a016234f46d24f48e1
paper_authors.name: Kangsan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:23.078Z
paper_authors._id: 66ed7737f2f27a5dfd81ef09
paper_authors.avatarUrl: /avatars/f45eea356e92ac7b3db23c2c92dec9fa.svg
paper_authors.fullname: Kangsan Kim
paper_authors.type: user
paper_authors.user: KangsanKim71
paper_authors._id: 674fc0a016234f46d24f48e2
paper_authors.name: Geon Park
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:48.617Z
paper_authors._id: 646cae3093badbc8c2e891c7
paper_authors.avatarUrl: /avatars/4aae2aca70ea9dc58dd6f9f9b2be15e1.svg
paper_authors.fullname: Geon Park
paper_authors.type: user
paper_authors.user: geonp
paper_authors._id: 674fc0a016234f46d24f48e3
paper_authors.name: Youngwan Lee
paper_authors._id: 674fc0a016234f46d24f48e4
paper_authors.name: Woongyeong Yeo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:45:38.836Z
paper_authors._id: 66d30f5fad293ffc4b7672bc
paper_authors.avatarUrl: /avatars/6f164d813b947940a088820f8fd4dbe8.svg
paper_authors.fullname: Woongyeong Yeo
paper_authors.type: user
paper_authors.user: wgcyeo
paper_authors._id: 674fc0a016234f46d24f48e5
paper_authors.name: Sung Ju Hwang
paper_title: VideoICL: Confidence-based Iterative In-context Learning for
    Out-of-Distribution Video Understanding
paper_summary: Recent advancements in video large multimodal models (LMMs) have
  significantly improved their video understanding and reasoning capabilities.
  However, their performance drops on out-of-distribution (OOD) tasks that are
  underrepresented in training data. Traditional methods like fine-tuning on OOD
  datasets are impractical due to high computational costs. While In-context
  learning (ICL) with demonstration examples has shown promising generalization
  performance in language tasks and image-language tasks without fine-tuning,
  applying ICL to video-language tasks faces challenges due to the limited
  context length in Video LMMs, as videos require longer token lengths. To
  address these issues, we propose VideoICL, a novel video in-context learning
  framework for OOD tasks that introduces a similarity-based relevant example
  selection strategy and a confidence-based iterative inference approach. This
  allows to select the most relevant examples and rank them based on similarity,
  to be used for inference. If the generated response has low confidence, our
  framework selects new examples and performs inference again, iteratively
  refining the results until a high-confidence response is obtained. This
  approach improves OOD video understanding performance by extending effective
  context length without incurring high costs. The experimental results on
  multiple benchmarks demonstrate significant performance gains, especially in
  domain-specific scenarios, laying the groundwork for broader video
  comprehension applications. Code will be released at
  https://github.com/KangsanKim07/VideoICL
paper_ai_keywords: multimodal models (LMMs) out-of-distribution (OOD) fine-tuning in-context learning (ICL) token lengths similarity-based relevant example selection confidence-based iterative inference VideoICL domain-specific scenarios video comprehension applications