Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations
<|reference_start|>Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations: In this study, we consider the problem of predicting task success for open-vocabulary manipulation by a manipulator, based on instruction sentences and egocentric images before and after manipulation. Conventional approaches, including multimodal large language models (MLLMs), often fail to appropriately understand detailed characteristics of objects and/or subtle changes in the position of objects. We propose Contrastive $\lambda$-Repformer, which predicts task success for table-top manipulation tasks by aligning images with instruction sentences. Our method integrates the following three key types of features into a multi-level aligned representation: features that preserve local image information; features aligned with natural language; and features structured through natural language. This allows the model to focus on important changes by looking at the differences in the representation between two images. We evaluate Contrastive $\lambda$-Repformer on a dataset based on a large-scale standard dataset, the RT-1 dataset, and on a physical robot platform. The results show that our approach outperformed existing approaches including MLLMs. Our best model achieved an improvement of 8.66 points in accuracy compared to the representative MLLM-based model.<|reference_end|>
@article{goko2024task,
    title={Task Success Prediction for Open-Vocabulary Manipulation Based on
  Multi-Level Aligned Representations},
    author={Miyu Goko, Motonari Kambara, Daichi Saito, Seitaro Otsuki, Komei
  Sugiura},
    journal={arXiv preprint arXiv:2410.00436},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00436},
    primaryClass={cs.RO cs.CV}
}