ICAL: Continual Learning of Multimodal Agents by Transforming
  Trajectories into Actionable Insights
Large-scale generative language and vision-language models (LLMs and VLMs)
excel in few-shot in-context learning for decision making and instruction
following. However, they require high-quality exemplar demonstrations to be
included in their context window. In this work, we ask: Can LLMs and VLMs
generate their own prompt examples from generic, sub-optimal demonstrations? We
propose In-Context Abstraction Learning (ICAL), a method that builds a memory
of multimodal experience insights from sub-optimal demonstrations and human
feedback. Given a noisy demonstration in a new domain, VLMs abstract the
trajectory into a general program by fixing inefficient actions and annotating
cognitive abstractions: task relationships, object state changes, temporal
subgoals, and task construals. These abstractions are refined and adapted
interactively through human feedback while the agent attempts to execute the
trajectory in a similar environment. The resulting abstractions, when used as
exemplars in the prompt, significantly improve decision-making in
retrieval-augmented LLM and VLM agents. Our ICAL agent surpasses the
state-of-the-art in dialogue-based instruction following in TEACh, multimodal
web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we
achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our
task success rate improves over the SOTA from 14.3% to 22.7%. In Ego4D action
forecasting, we improve over few-shot GPT-4V and remain competitive with
supervised models. We show finetuning our retrieval-augmented in-context agent
yields additional improvements. Our approach significantly reduces reliance on
expert-crafted examples and consistently outperforms in-context learning from
action plans that lack such insights.
6679c53e3463f1209e2f9c19
Gabriel Sarch
claimed_verified
2024-06-26T08:05:08.809Z
6666828b15ec3fd28b52a1bd
https://cdn-avatars.huggingface.co/v1/production/uploads/6666828b15ec3fd28b52a1bd/jRPAQmvg_J6J84Louar6P.jpeg
Gabriel H Sarch
user
gsarch
6679c53e3463f1209e2f9c1a
Lawrence Jang
claimed_verified
2024-09-13T07:21:06.325Z
664aebe829eadb3ab4e4ca3f
/avatars/548d5656e082c8959ac78b883f0805af.svg
Lawrence Jang
user
ljang0
6679c53e3463f1209e2f9c1b
Michael J. Tarr
6679c53e3463f1209e2f9c1c
William W. Cohen
6679c53e3463f1209e2f9c1d
Kenneth Marino
6679c53e3463f1209e2f9c1e
Katerina Fragkiadaki
ICAL: Continual Learning of Multimodal Agents by Transforming
  Trajectories into Actionable Insights
Large-scale generative language and vision-language models (LLMs and VLMs)
excel in few-shot in-context learning for decision making and instruction
following. However, they require high-quality exemplar demonstrations to be
included in their context window. In this work, we ask: Can LLMs and VLMs
generate their own prompt examples from generic, sub-optimal demonstrations? We
propose In-Context Abstraction Learning (ICAL), a method that builds a memory
of multimodal experience insights from sub-optimal demonstrations and human
feedback. Given a noisy demonstration in a new domain, VLMs abstract the
trajectory into a general program by fixing inefficient actions and annotating
cognitive abstractions: task relationships, object state changes, temporal
subgoals, and task construals. These abstractions are refined and adapted
interactively through human feedback while the agent attempts to execute the
trajectory in a similar environment. The resulting abstractions, when used as
exemplars in the prompt, significantly improve decision-making in
retrieval-augmented LLM and VLM agents. Our ICAL agent surpasses the
state-of-the-art in dialogue-based instruction following in TEACh, multimodal
web agents in VisualWebArena, and action anticipation in Ego4D. In TEACh, we
achieve a 12.6% improvement in goal-condition success. In VisualWebArena, our
task success rate improves over the SOTA from 14.3% to 22.7%. In Ego4D action
forecasting, we improve over few-shot GPT-4V and remain competitive with
supervised models. We show finetuning our retrieval-augmented in-context agent
yields additional improvements. Our approach significantly reduces reliance on
expert-crafted examples and consistently outperforms in-context learning from
action plans that lack such insights.
In-Context Abstraction Learning (ICAL) improves decision-making in LLM and VLM agents by generating high-quality prompt examples from sub-optimal demonstrations and human feedback.
LLMs VLMs few-shot in-context learning In-Context Abstraction Learning ICAL multimodal experience insights noisy demonstration trajectory general program inefficient actions cognitive abstractions task relationships object state changes temporal subgoals task construals human feedback retrieval-augmented agents dialogue-based instruction following multimodal web agents action anticipation goal-condition success task success rate action forecasting few-shot GPT-4V supervised models finetuning