SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy Adaptation
<|reference_start|>SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy Adaptation: This work explores the zero-shot adaptation capability of semantic skills, semantically interpretable experts' behavior patterns, in cross-domain settings, where a user input in interleaved multi-modal snippets can prompt a new long-horizon task for different domains. In these cross-domain settings, we present a semantic skill translator framework SemTra which utilizes a set of multi-modal models to extract skills from the snippets, and leverages the reasoning capabilities of a pretrained language model to adapt these extracted skills to the target domain. The framework employs a two-level hierarchy for adaptation: task adaptation and skill adaptation. During task adaptation, seq-to-seq translation by the language model transforms the extracted skills into a semantic skill sequence, which is tailored to fit the cross-domain contexts. Skill adaptation focuses on optimizing each semantic skill for the target domain context, through parametric instantiations that are facilitated by language prompting and contrastive learning-based context inferences. This hierarchical adaptation empowers the framework to not only infer a complex task specification in one-shot from the interleaved multi-modal snippets, but also adapt it to new domains with zero-shot learning abilities. We evaluate our framework with Meta-World, Franka Kitchen, RLBench, and CARLA environments. The results clarify the framework's superiority in performing long-horizon tasks and adapting to different domains, showing its broad applicability in practical use cases, such as cognitive robots interpreting abstract instructions and autonomous vehicles operating under varied configurations.<|reference_end|>
@article{shin2024semtra:,
    title={SemTra: A Semantic Skill Translator for Cross-Domain Zero-Shot Policy
  Adaptation},
    author={Sangwoo Shin, Minjong Yoo, Jeongwoo Lee, Honguk Woo},
    journal={arXiv preprint arXiv:2402.07418},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.07418},
    primaryClass={cs.AI}
}