title: LLaNA: Large Language and NeRF Assistant
summary: Multimodal Large Language Models (MLLMs) have demonstrated an excellent
  understanding of images and 3D data. However, both modalities have shortcomings
  in holistically capturing the appearance and geometry of objects. Meanwhile,
  Neural Radiance Fields (NeRFs), which encode information within the weights of
  a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
  widespread modality that simultaneously encodes the geometry and photorealistic
  appearance of objects. This paper investigates the feasibility and
  effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
  general-purpose NeRF-language assistant capable of performing new tasks such as
  NeRF captioning and Q\&A. Notably, our method directly processes the weights of
  the NeRF's MLP to extract information about the represented objects without the
  need to render images or materialize 3D data structures. Moreover, we build a
  dataset of NeRFs with text annotations for various NeRF-language tasks with no
  human intervention. Based on this dataset, we develop a benchmark to evaluate
  the NeRF understanding capability of our method. Results show that processing
  NeRF weights performs favourably against extracting 2D or 3D representations
  from NeRFs.
paper_authors._id: 66713e14bb4aa4b6a9c83093
paper_authors.name: Andrea Amaduzzi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T08:29:35.846Z
paper_authors._id: 629f5f61ebbc7168c29f27c6
paper_authors.avatarUrl: /avatars/1065d941089dd3c817702f62295c4431.svg
paper_authors.fullname: Andrea
paper_authors.type: user
paper_authors.user: andreamaduzzi
paper_authors._id: 66713e14bb4aa4b6a9c83094
paper_authors.name: Pierluigi Zama Ramirez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:57:34.514Z
paper_authors._id: 651ea108fb142cd617e0aadf
paper_authors.avatarUrl: /avatars/0c0cbf5b66388c088e7c654222db8bf9.svg
paper_authors.fullname: Pierluigi Zama Ramirez
paper_authors.type: user
paper_authors.user: pierlui92
paper_authors._id: 66713e14bb4aa4b6a9c83095
paper_authors.name: Giuseppe Lisanti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-03T10:36:35.797Z
paper_authors._id: 6672a82e2a4d2e3e835f1f57
paper_authors.avatarUrl: /avatars/520a11f13d30272557c85e20aafb1922.svg
paper_authors.fullname: Giuseppe Lisanti
paper_authors.type: user
paper_authors.user: giuseppelisanti
paper_authors._id: 66713e14bb4aa4b6a9c83096
paper_authors.name: Samuele Salti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-19T09:43:38.468Z
paper_authors._id: 6672a802a3dc375df3fdfa3a
paper_authors.avatarUrl: /avatars/4f37c3fd80080d1c677ebc4a1c565f11.svg
paper_authors.fullname: Samuele Salti
paper_authors.type: user
paper_authors.user: samuelesalti
paper_authors._id: 66713e14bb4aa4b6a9c83097
paper_authors.name: Luigi Di Stefano
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-20T06:34:45.013Z
paper_authors._id: 6673cd4056cf3c3548349f16
paper_authors.avatarUrl: /avatars/71337fb4bad4e89d8fb8584535017cac.svg
paper_authors.fullname: Luigi Di Stefano
paper_authors.type: user
paper_authors.user: ldistef
paper_title: LLaNA: Large Language and NeRF Assistant
paper_summary: Multimodal Large Language Models (MLLMs) have demonstrated an excellent
  understanding of images and 3D data. However, both modalities have shortcomings
  in holistically capturing the appearance and geometry of objects. Meanwhile,
  Neural Radiance Fields (NeRFs), which encode information within the weights of
  a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly
  widespread modality that simultaneously encodes the geometry and photorealistic
  appearance of objects. This paper investigates the feasibility and
  effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first
  general-purpose NeRF-language assistant capable of performing new tasks such as
  NeRF captioning and Q\&A. Notably, our method directly processes the weights of
  the NeRF's MLP to extract information about the represented objects without the
  need to render images or materialize 3D data structures. Moreover, we build a
  dataset of NeRFs with text annotations for various NeRF-language tasks with no
  human intervention. Based on this dataset, we develop a benchmark to evaluate
  the NeRF understanding capability of our method. Results show that processing
  NeRF weights performs favourably against extracting 2D or 3D representations
  from NeRFs.
paper_ai_summary: LLaNA, an MLLM integrated with NeRFs, effectively performs tasks such as NeRF captioning and Q&A by processing NeRF weights directly, outperforming 2D or 3D representation extraction methods.
paper_ai_keywords: Multimodal Large Language Models MLLMs Neural Radiance Fields NeRFs Multi-Layer Perceptron MLP NeRF captioning NeRF Q&A benchmark NeRF understanding capability