title: Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards
    for Visuomotor Robot Policy Alignment
summary: Visuomotor robot policies, increasingly pre-trained on large-scale datasets,
  promise significant advancements across robotics domains. However, aligning
  these policies with end-user preferences remains a challenge, particularly when
  the preferences are hard to specify. While reinforcement learning from human
  feedback (RLHF) has become the predominant mechanism for alignment in
  non-embodied domains like large language models, it has not seen the same
  success in aligning visuomotor policies due to the prohibitive amount of human
  feedback required to learn visual reward functions. To address this limitation,
  we propose Representation-Aligned Preference-based Learning (RAPL), an
  observation-only method for learning visual rewards from significantly less
  human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback
  on fine-tuning pre-trained vision encoders to align with the end-user's visual
  representation and then constructs a dense visual reward via feature matching
  in this aligned representation space. We first validate RAPL through simulation
  experiments in the X-Magical benchmark and Franka Panda robotic manipulation,
  demonstrating that it can learn rewards aligned with human preferences, more
  efficiently uses preference data, and generalizes across robot embodiments.
  Finally, our hardware experiments align pre-trained Diffusion Policies for
  three object manipulation tasks. We find that RAPL can fine-tune these policies
  with 5x less real human preference data, taking the first step towards
  minimizing human feedback while maximizing visuomotor robot policy alignment.
paper_authors._id: 67595a8662950d6eeb940516
paper_authors.name: Ran Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T08:22:35.613Z
paper_authors._id: 6535786a24c3063696804cdf
paper_authors.avatarUrl: /avatars/2e59a26b691a9ab9825a015c7d47f9aa.svg
paper_authors.fullname: Ran (Thomas) Tian
paper_authors.type: user
paper_authors.user: thomasrantian
paper_authors._id: 67595a8662950d6eeb940517
paper_authors.name: Yilin Wu
paper_authors._id: 67595a8662950d6eeb940518
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T14:03:46.829Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 67595a8662950d6eeb940519
paper_authors.name: Masayoshi Tomizuka
paper_authors._id: 67595a8662950d6eeb94051a
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T14:03:37.137Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_authors._id: 67595a8662950d6eeb94051b
paper_authors.name: Andrea Bajcsy
paper_title: Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards
    for Visuomotor Robot Policy Alignment
paper_summary: Visuomotor robot policies, increasingly pre-trained on large-scale datasets,
  promise significant advancements across robotics domains. However, aligning
  these policies with end-user preferences remains a challenge, particularly when
  the preferences are hard to specify. While reinforcement learning from human
  feedback (RLHF) has become the predominant mechanism for alignment in
  non-embodied domains like large language models, it has not seen the same
  success in aligning visuomotor policies due to the prohibitive amount of human
  feedback required to learn visual reward functions. To address this limitation,
  we propose Representation-Aligned Preference-based Learning (RAPL), an
  observation-only method for learning visual rewards from significantly less
  human preference feedback. Unlike traditional RLHF, RAPL focuses human feedback
  on fine-tuning pre-trained vision encoders to align with the end-user's visual
  representation and then constructs a dense visual reward via feature matching
  in this aligned representation space. We first validate RAPL through simulation
  experiments in the X-Magical benchmark and Franka Panda robotic manipulation,
  demonstrating that it can learn rewards aligned with human preferences, more
  efficiently uses preference data, and generalizes across robot embodiments.
  Finally, our hardware experiments align pre-trained Diffusion Policies for
  three object manipulation tasks. We find that RAPL can fine-tune these policies
  with 5x less real human preference data, taking the first step towards
  minimizing human feedback while maximizing visuomotor robot policy alignment.
paper_ai_keywords: reinforcement learning from human feedback (RLHF) pre-trained vision encoders visual reward functions feature matching X-Magical benchmark Franka Panda robotic manipulation real human preference feedback Interestusion Policies visuomotor robot policies pre-trained policies fine-tuning