title: ICAL: Continual Learning of Multimodal Agents by Transforming
    Trajectories into Actionable Insights
summary: Large-scale generative language and vision-language models (LLMs and VLMs)
  excel in few-shot in-context learning for decision making and instruction
  following. However, they require high-quality exemplar demonstrations to be
  included in their context window. In this work, we ask: Can LLMs and VLMs
  generate their own prompt examples from generic, sub-optimal demonstrations? We
  propose In-Context Abstraction Learning (ICAL), a method that builds a memory
  of multimodal experience insights from sub-optimal demonstrations and human
  feedback. Given a noisy demonstration in a new domain, VLMs abstract the
  trajectory into a general program by fixing inefficient actions and annotating
  cognitive abstractions: task relationships, object state changes, temporal
  subgoals, and task construals. These abstractions are refined and adapted
  interactively through human feedback while the agent attempts to execute the
  trajectory in a similar environment. The resulting abstractions, when used as
  exemplars in the prompt, significantly improve decision-making in
  retrieval-augmented LLM and VLM agents. Our ICAL agent surpasses the
  state-of-the-art in dialogue-based instruction following in TEACh, multimodal
  web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we
  achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our
  task success rate improves over the SOTA from 14.3% to 22.7%. In Ego4D action
  forecasting, we improve over few-shot GPT-4V and remain competitive with
  supervised models. We show finetuning our retrieval-augmented in-context agent
  yields additional improvements. Our approach significantly reduces reliance on
  expert-crafted examples and consistently outperforms in-context learning from
  action plans that lack such insights.
paper_authors._id: 6679c53e3463f1209e2f9c19
paper_authors.name: Gabriel Sarch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:08.809Z
paper_authors._id: 6666828b15ec3fd28b52a1bd
paper_authors.avatarUrl: /avatars/a80eeac5dd23aabce1eb299ebe7f245b.svg
paper_authors.fullname: Gabriel H Sarch
paper_authors.type: user
paper_authors.user: gsarch
paper_authors._id: 6679c53e3463f1209e2f9c1a
paper_authors.name: Lawrence Jang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T07:21:06.325Z
paper_authors._id: 664aebe829eadb3ab4e4ca3f
paper_authors.avatarUrl: /avatars/548d5656e082c8959ac78b883f0805af.svg
paper_authors.fullname: Lawrence Jang
paper_authors.type: user
paper_authors.user: ljang0
paper_authors._id: 6679c53e3463f1209e2f9c1b
paper_authors.name: Michael J. Tarr
paper_authors._id: 6679c53e3463f1209e2f9c1c
paper_authors.name: William W. Cohen
paper_authors._id: 6679c53e3463f1209e2f9c1d
paper_authors.name: Kenneth Marino
paper_authors._id: 6679c53e3463f1209e2f9c1e
paper_authors.name: Katerina Fragkiadaki
paper_title: ICAL: Continual Learning of Multimodal Agents by Transforming
    Trajectories into Actionable Insights
paper_summary: Large-scale generative language and vision-language models (LLMs and VLMs)
  excel in few-shot in-context learning for decision making and instruction
  following. However, they require high-quality exemplar demonstrations to be
  included in their context window. In this work, we ask: Can LLMs and VLMs
  generate their own prompt examples from generic, sub-optimal demonstrations? We
  propose In-Context Abstraction Learning (ICAL), a method that builds a memory
  of multimodal experience insights from sub-optimal demonstrations and human
  feedback. Given a noisy demonstration in a new domain, VLMs abstract the
  trajectory into a general program by fixing inefficient actions and annotating
  cognitive abstractions: task relationships, object state changes, temporal
  subgoals, and task construals. These abstractions are refined and adapted
  interactively through human feedback while the agent attempts to execute the
  trajectory in a similar environment. The resulting abstractions, when used as
  exemplars in the prompt, significantly improve decision-making in
  retrieval-augmented LLM and VLM agents. Our ICAL agent surpasses the
  state-of-the-art in dialogue-based instruction following in TEACh, multimodal
  web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we
  achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our
  task success rate improves over the SOTA from 14.3% to 22.7%. In Ego4D action
  forecasting, we improve over few-shot GPT-4V and remain competitive with
  supervised models. We show finetuning our retrieval-augmented in-context agent
  yields additional improvements. Our approach significantly reduces reliance on
  expert-crafted examples and consistently outperforms in-context learning from
  action plans that lack such insights.
paper_ai_summary: In-Context Abstraction Learning (ICAL) improves decision-making in LLM and VLM agents by generating high-quality prompt examples from sub-optimal demonstrations and human feedback.
paper_ai_keywords: LLMs VLMs few-shot in-context learning In-Context Abstraction Learning ICAL multimodal experience insights noisy demonstration trajectory general program inefficient actions cognitive abstractions task relationships object state changes temporal subgoals task construals human feedback retrieval-augmented agents dialogue-based instruction following multimodal web agents action anticipation goal-condition success task success rate action forecasting few-shot GPT-4V supervised models finetuning