Embodied Executable Policy Learning with Language-based Scene
  Summarization
Large Language models (LLMs) have shown remarkable success in assisting robot
learning tasks, i.e., complex household planning. However, the performance of
pretrained LLMs heavily relies on domain-specific templated text data, which
may be infeasible in real-world robot learning tasks with image-based
observations. Moreover, existing LLMs with text inputs lack the capability to
evolve with non-expert interactions with environments. In this work, we
introduce a novel learning paradigm that generates robots' executable actions
in the form of text, derived solely from visual observations, using
language-based summarization of these observations as the connecting bridge
between both domains. Our proposed paradigm stands apart from previous works,
which utilized either language instructions or a combination of language and
visual data as inputs. Moreover, our method does not require oracle text
summarization of the scene, eliminating the need for human involvement in the
learning loop, which makes it more practical for real-world robot learning
tasks. Our proposed paradigm consists of two modules: the SUM module, which
interprets the environment using visual observations and produces a text
summary of the scene, and the APM module, which generates executable action
policies based on the natural language descriptions provided by the SUM module.
We demonstrate that our proposed method can employ two fine-tuning strategies,
including imitation learning and reinforcement learning approaches, to adapt to
the target test tasks effectively. We conduct extensive experiments involving
various SUM/APM model selections, environments, and tasks across 7 house
layouts in the VirtualHome environment. Our experimental results demonstrate
that our method surpasses existing baselines, confirming the effectiveness of
this novel learning paradigm.
64866eb1216927cc9cf28db0
Jielin Qiu
claimed_verified
2023-06-12T18:59:08.304Z
63294eeace50f125375243c1
/avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
Jielin Qiu
user
jasonqiu
64866eb1216927cc9cf28db1
Mengdi Xu
admin_assigned
2023-06-13T15:06:42.093Z
639187d92e1b430e96d45bc6
/avatars/1e1ee95a75adc4145166c5b50cbd524f.svg
Mengdi Xu
user
mengdi
64866eb1216927cc9cf28db2
William Han
claimed_verified
2025-02-28T15:15:03.458Z
65e50e9e72e748aae5823dd3
/avatars/216cf665ca91a70e4a7aa2814654f4e4.svg
William Han
user
willxxy
64866eb1216927cc9cf28db3
Seungwhan Moon
extracted_pending
2023-08-02T01:16:07.001Z
64c9ae24e761f47061448bfb
/avatars/8835bcf769fd738e8c66fa7a52ddc421.svg
Seungwhan Moon
user
shanemoon27
64866eb1216927cc9cf28db4
Ding Zhao
Embodied Executable Policy Learning with Language-based Scene
  Summarization
Large Language models (LLMs) have shown remarkable success in assisting robot
learning tasks, i.e., complex household planning. However, the performance of
pretrained LLMs heavily relies on domain-specific templated text data, which
may be infeasible in real-world robot learning tasks with image-based
observations. Moreover, existing LLMs with text inputs lack the capability to
evolve with non-expert interactions with environments. In this work, we
introduce a novel learning paradigm that generates robots' executable actions
in the form of text, derived solely from visual observations, using
language-based summarization of these observations as the connecting bridge
between both domains. Our proposed paradigm stands apart from previous works,
which utilized either language instructions or a combination of language and
visual data as inputs. Moreover, our method does not require oracle text
summarization of the scene, eliminating the need for human involvement in the
learning loop, which makes it more practical for real-world robot learning
tasks. Our proposed paradigm consists of two modules: the SUM module, which
interprets the environment using visual observations and produces a text
summary of the scene, and the APM module, which generates executable action
policies based on the natural language descriptions provided by the SUM module.
We demonstrate that our proposed method can employ two fine-tuning strategies,
including imitation learning and reinforcement learning approaches, to adapt to
the target test tasks effectively. We conduct extensive experiments involving
various SUM/APM model selections, environments, and tasks across 7 house
layouts in the VirtualHome environment. Our experimental results demonstrate
that our method surpasses existing baselines, confirming the effectiveness of
this novel learning paradigm.
A novel learning paradigm generates robot actions from visual observations using language-based scene summarization, outperforming existing methods in real-world robot learning tasks.
Large Language Models LLMs robot learning household planning domain-specific templated text data image-based observations language-based summarization executable actions SUM module APM module imitation learning reinforcement learning VirtualHome environment