title: Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
    Realistic Environments
summary: Autonomous agents powered by large language models (LLMs) have the potential
  to enhance human capabilities, assisting with digital tasks from sending emails
  to performing data analysis. The abilities of existing LLMs at such tasks are
  often hindered by the lack of high-quality agent data from the corresponding
  environments they interact with. We propose Learn-by-interact, a data-centric
  framework to adapt LLM agents to any given environments without human
  annotations. Learn-by-interact synthesizes trajectories of agent-environment
  interactions based on documentations, and constructs instructions by
  summarizing or abstracting the interaction histories, a process called backward
  construction. We assess the quality of our synthetic data by using them in both
  training-based scenarios and training-free in-context learning (ICL), where we
  craft innovative retrieval approaches optimized for agents. Extensive
  experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across
  realistic coding, web, and desktop environments show the effectiveness of
  Learn-by-interact in various downstream agentic tasks -- baseline results are
  improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with
  Codestral-22B. We further demonstrate the critical role of backward
  construction, which provides up to 14.0\% improvement for training. Our
  ablation studies demonstrate the efficiency provided by our synthesized data in
  ICL and the superiority of our retrieval pipeline over alternative approaches
  like conventional retrieval-augmented generation (RAG). We expect that
  Learn-by-interact will serve as a foundation for agent data synthesis as LLMs
  are increasingly deployed at real-world environments.
paper_authors._id: 67907dd5e1d8fc832b3e7b0f
paper_authors.name: Hongjin Su
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:51:36.858Z
paper_authors._id: 632d53951538d4798a73c849
paper_authors.avatarUrl: /avatars/b7d0a895e669bcd1303c4716b5401c36.svg
paper_authors.fullname: Hongjin SU
paper_authors.type: user
paper_authors.user: multi-train
paper_authors._id: 67907dd5e1d8fc832b3e7b10
paper_authors.name: Ruoxi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:51:57.105Z
paper_authors._id: 653093ad471261a41241e048
paper_authors.avatarUrl: /avatars/54263ce75b102b81211d27b696920870.svg
paper_authors.fullname: Ruoxi Sun
paper_authors.type: user
paper_authors.user: brycesun
paper_authors._id: 67907dd5e1d8fc832b3e7b11
paper_authors.name: Jinsung Yoon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:52:16.664Z
paper_authors._id: 64da5c05322a5774e085339a
paper_authors.avatarUrl: /avatars/ef6a4325622799d474af778b92232555.svg
paper_authors.fullname: Jinsung Yoon
paper_authors.type: user
paper_authors.user: jsyoon0823
paper_authors._id: 67907dd5e1d8fc832b3e7b12
paper_authors.name: Pengcheng Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-22T15:52:34.099Z
paper_authors._id: 65024921fa0dccbd859e55e9
paper_authors.avatarUrl: /avatars/1f3c31c21ca692c2a63cf3437eac2507.svg
paper_authors.fullname: Pengcheng Yin
paper_authors.type: user
paper_authors.user: pchyin
paper_authors._id: 67907dd5e1d8fc832b3e7b13
paper_authors.name: Tao Yu
paper_authors._id: 67907dd5e1d8fc832b3e7b14
paper_authors.name: Sercan Ö. Arık
paper_title: Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
    Realistic Environments
paper_summary: Autonomous agents powered by large language models (LLMs) have the potential
  to enhance human capabilities, assisting with digital tasks from sending emails
  to performing data analysis. The abilities of existing LLMs at such tasks are
  often hindered by the lack of high-quality agent data from the corresponding
  environments they interact with. We propose Learn-by-interact, a data-centric
  framework to adapt LLM agents to any given environments without human
  annotations. Learn-by-interact synthesizes trajectories of agent-environment
  interactions based on documentations, and constructs instructions by
  summarizing or abstracting the interaction histories, a process called backward
  construction. We assess the quality of our synthetic data by using them in both
  training-based scenarios and training-free in-context learning (ICL), where we
  craft innovative retrieval approaches optimized for agents. Extensive
  experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across
  realistic coding, web, and desktop environments show the effectiveness of
  Learn-by-interact in various downstream agentic tasks -- baseline results are
  improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with
  Codestral-22B. We further demonstrate the critical role of backward
  construction, which provides up to 14.0\% improvement for training. Our
  ablation studies demonstrate the efficiency provided by our synthesized data in
  ICL and the superiority of our retrieval pipeline over alternative approaches
  like conventional retrieval-augmented generation (RAG). We expect that
  Learn-by-interact will serve as a foundation for agent data synthesis as LLMs
  are increasingly deployed at real-world environments.
paper_ai_keywords: Learn-by-interact documentations backward construction in-context learning ICL robust retrieval approaches SWE-bench WebArena OSWorld Spider2-V steeric tasks Claude-3.5 Codestral-22B conventional retrieval-augmented generation RAG