title: Embodied Executable Policy Learning with Language-based Scene
    Summarization
summary: Large Language models (LLMs) have shown remarkable success in assisting robot
  learning tasks, i.e., complex household planning. However, the performance of
  pretrained LLMs heavily relies on domain-specific templated text data, which
  may be infeasible in real-world robot learning tasks with image-based
  observations. Moreover, existing LLMs with text inputs lack the capability to
  evolve with non-expert interactions with environments. In this work, we
  introduce a novel learning paradigm that generates robots' executable actions
  in the form of text, derived solely from visual observations, using
  language-based summarization of these observations as the connecting bridge
  between both domains. Our proposed paradigm stands apart from previous works,
  which utilized either language instructions or a combination of language and
  visual data as inputs. Moreover, our method does not require oracle text
  summarization of the scene, eliminating the need for human involvement in the
  learning loop, which makes it more practical for real-world robot learning
  tasks. Our proposed paradigm consists of two modules: the SUM module, which
  interprets the environment using visual observations and produces a text
  summary of the scene, and the APM module, which generates executable action
  policies based on the natural language descriptions provided by the SUM module.
  We demonstrate that our proposed method can employ two fine-tuning strategies,
  including imitation learning and reinforcement learning approaches, to adapt to
  the target test tasks effectively. We conduct extensive experiments involving
  various SUM/APM model selections, environments, and tasks across 7 house
  layouts in the VirtualHome environment. Our experimental results demonstrate
  that our method surpasses existing baselines, confirming the effectiveness of
  this novel learning paradigm.
paper_authors._id: 64866eb1216927cc9cf28db0
paper_authors.name: Jielin Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T18:59:08.304Z
paper_authors._id: 63294eeace50f125375243c1
paper_authors.avatarUrl: /avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
paper_authors.fullname: Jielin Qiu
paper_authors.type: user
paper_authors.user: jasonqiu
paper_authors._id: 64866eb1216927cc9cf28db1
paper_authors.name: Mengdi Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T15:06:42.093Z
paper_authors._id: 639187d92e1b430e96d45bc6
paper_authors.avatarUrl: /avatars/1e1ee95a75adc4145166c5b50cbd524f.svg
paper_authors.fullname: Mengdi Xu
paper_authors.type: user
paper_authors.user: mengdi
paper_authors._id: 64866eb1216927cc9cf28db2
paper_authors.name: William Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T15:15:03.458Z
paper_authors._id: 65e50e9e72e748aae5823dd3
paper_authors.avatarUrl: /avatars/216cf665ca91a70e4a7aa2814654f4e4.svg
paper_authors.fullname: William Han
paper_authors.type: user
paper_authors.user: willxxy
paper_authors._id: 64866eb1216927cc9cf28db3
paper_authors.name: Seungwhan Moon
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-08-02T01:16:07.001Z
paper_authors._id: 64c9ae24e761f47061448bfb
paper_authors.avatarUrl: /avatars/8835bcf769fd738e8c66fa7a52ddc421.svg
paper_authors.fullname: Seungwhan Moon
paper_authors.type: user
paper_authors.user: shanemoon27
paper_authors._id: 64866eb1216927cc9cf28db4
paper_authors.name: Ding Zhao
paper_title: Embodied Executable Policy Learning with Language-based Scene
    Summarization
paper_summary: Large Language models (LLMs) have shown remarkable success in assisting robot
  learning tasks, i.e., complex household planning. However, the performance of
  pretrained LLMs heavily relies on domain-specific templated text data, which
  may be infeasible in real-world robot learning tasks with image-based
  observations. Moreover, existing LLMs with text inputs lack the capability to
  evolve with non-expert interactions with environments. In this work, we
  introduce a novel learning paradigm that generates robots' executable actions
  in the form of text, derived solely from visual observations, using
  language-based summarization of these observations as the connecting bridge
  between both domains. Our proposed paradigm stands apart from previous works,
  which utilized either language instructions or a combination of language and
  visual data as inputs. Moreover, our method does not require oracle text
  summarization of the scene, eliminating the need for human involvement in the
  learning loop, which makes it more practical for real-world robot learning
  tasks. Our proposed paradigm consists of two modules: the SUM module, which
  interprets the environment using visual observations and produces a text
  summary of the scene, and the APM module, which generates executable action
  policies based on the natural language descriptions provided by the SUM module.
  We demonstrate that our proposed method can employ two fine-tuning strategies,
  including imitation learning and reinforcement learning approaches, to adapt to
  the target test tasks effectively. We conduct extensive experiments involving
  various SUM/APM model selections, environments, and tasks across 7 house
  layouts in the VirtualHome environment. Our experimental results demonstrate
  that our method surpasses existing baselines, confirming the effectiveness of
  this novel learning paradigm.
paper_ai_keywords: language-based summarization executable actions natural language descriptions imitation learning reinforcement learning VirtualHome environment