PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
<|reference_start|>PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning: Preference-based reinforcement learning (RL) has emerged as a new field in robot learning, where humans play a pivotal role in shaping robot behavior by expressing preferences on different sequences of state-action pairs. However, formulating realistic policies for robots demands responses from humans to an extensive array of queries. In this work, we approach the sample-efficiency challenge by expanding the information collected per query to contain both preferences and optional text prompting. To accomplish this, we leverage the zero-shot capabilities of a large language model (LLM) to reason from the text provided by humans. To accommodate the additional query information, we reformulate the reward learning objectives to contain flexible highlights -- state-action pairs that contain relatively high information and are related to the features processed in a zero-shot fashion from a pretrained LLM. In both a simulated scenario and a user study, we reveal the effectiveness of our work by analyzing the feedback and its implications. Additionally, the collective feedback collected serves to train a robot on socially compliant trajectories in a simulated social navigation landscape. We provide video examples of the trained policies at https://sites.google.com/view/rl-predilect<|reference_end|>
@article{holk2024predilect:,
    title={PREDILECT: Preferences Delineated with Zero-Shot Language-based
  Reasoning in Reinforcement Learning},
    author={Simon Holk, Daniel Marta, Iolanda Leite},
    journal={arXiv preprint arXiv:2402.15420},
    year={2024},
    doi={10.1145/3610977.3634970},
    archivePrefix={arXiv},
    eprint={2402.15420},
    primaryClass={cs.RO cs.CL cs.LG}
}