readme: ---
  license: mit
  ---
  
  # Model Card
  
  ## Model Details
  VIMA (**Vi**suo**M**otor **A**ttention, reads "_v-eye-ma_") is a novel Transformer model that ingests multimodal prompts and outputs robot arm control autoregressively. VIMA is developed primarily by researchers at NVIDIA.
  
  ### Model Date
  October 2022
  
  ### Model Type
  VIMA model consists of a pretrained T5 model as the prompt encoder, several tokenizers to process multimodal inputs, and a causal decoder that autoregressively predicts actions given the prompt and interaction history.
  
  ### Model Versions
  We released 7 checkpoints covering a spectrum of model capacity from 2M to 200M.
  
  ## Model Use
  
  ### Intended Use
  The model is intended to be used alongside [VIMA-Bench](https://github.com/vimalabs/VimaBench) to study general robot manipulation with multimodal prompts.
  
  
  ### Primary intended uses
  The primary intended users of these models are AI researchers in robotics, multimodal learning, embodied agents, foundation models, etc.
  
  ## Data
  The models were trained with [data](https://doi.org/10.5281/zenodo.7127587) generated by oracles implemented in [VIMA-Bench](https://github.com/vimalabs/VimaBench). It includes 650K successful trajectories for behavior cloning. We use 600K trajectories for training. The remaining 50K trajectories are held out for validation purpose.
  
  ## Performance and Limitations
  ### Metrics and Performance
  We quantify the performance of trained models using task success percentage aggregated over multiple tasks. We evaluate models' performance on task suite from [VIMA-Bench](https://github.com/vimalabs/VimaBench) and follow the proposed evaluation protocol. See our paper for more details.
  
  ### Limitations
  Our provided model checkpoints are pretrained on VIMA-Bench, which may not directly generalize to other simulators or real world. Techniques like [SECANT](https://linxifan.github.io/secant-site/) can be applied to enable sim2real transfer.
  
  ## Paper and Citation
  
  Our paper is posted on [arXiv](https://arxiv.org/abs/2210.03094). If you find our work useful, please consider citing us! 
  
  ```bibtex
  @article{jiang2022vima,
    title   = {VIMA: General Robot Manipulation with Multimodal Prompts},
    author  = {Yunfan Jiang and Agrim Gupta and Zichen Zhang and Guanzhi Wang and Yongqiang Dou and Yanjun Chen and Li Fei-Fei and Anima Anandkumar and Yuke Zhu and Linxi Fan},
    year    = {2022},
    journal = {arXiv preprint arXiv: Arxiv-2210.03094}
  }
  ```