Connecting NeRFs, Images, and Text
<|reference_start|>Connecting NeRFs, Images, and Text: Neural Radiance Fields (NeRFs) have emerged as a standard framework for representing 3D scenes and objects, introducing a novel data type for information exchange and storage. Concurrently, significant progress has been made in multimodal representation learning for text and image data. This paper explores a novel research direction that aims to connect the NeRF modality with other modalities, similar to established methodologies for images and text. To this end, we propose a simple framework that exploits pre-trained models for NeRF representations alongside multimodal models for text and image processing. Our framework learns a bidirectional mapping between NeRF embeddings and those obtained from corresponding images and text. This mapping unlocks several novel and useful applications, including NeRF zero-shot classification and NeRF retrieval from images or text.<|reference_end|>
@article{ballerini2024connecting,
    title={Connecting NeRFs, Images, and Text},
    author={Francesco Ballerini, Pierluigi Zama Ramirez, Roberto Mirabella,
  Samuele Salti, Luigi Di Stefano},
    journal={arXiv preprint arXiv:2404.07993},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.07993},
    primaryClass={cs.CV}
}