LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models
<|reference_start|>LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models: When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully reason about the deformation behavior of elasto-plastic objects. Furthermore, we find that LLM-Craft is able to successfully create a set of simple letter shapes. Finally, we explore extending the framework to reaching more ambiguous semantic goals, such as "thinner" or "bumpy". For videos please see our website: https://sites.google.com/andrew.cmu.edu/llmcraft.<|reference_end|>
@article{bartsch2024llm-craft:,
    title={LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large
  Language Models},
    author={Alison Bartsch and Amir Barati Farimani},
    journal={arXiv preprint arXiv:2406.08648},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.08648},
    primaryClass={cs.RO}
}