title: Exploring the Potential of Encoder-free Architectures in 3D LMMs
summary: Encoder-free architectures have been preliminarily explored in the 2D visual
  domain, yet it remains an open question whether they can be effectively applied
  to 3D understanding scenarios. In this paper, we present the first
  comprehensive investigation into the potential of encoder-free architectures to
  overcome the challenges of encoder-based 3D Large Multimodal Models (LMMs).
  These challenges include the failure to adapt to varying point cloud
  resolutions and the point features from the encoder not meeting the semantic
  needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to
  remove the encoder and enable the LLM to assume the role of the 3D encoder: 1)
  We propose the LLM-embedded Semantic Encoding strategy in the pre-training
  stage, exploring the effects of various point cloud self-supervised losses. And
  we present the Hybrid Semantic Loss to extract high-level semantics. 2) We
  introduce the Hierarchical Geometry Aggregation strategy in the instruction
  tuning stage. This incorporates inductive bias into the LLM early layers to
  focus on the local details of the point clouds. To the end, we present the
  first Encoder-free 3D LMM, ENEL. Our 7B model rivals the current
  state-of-the-art model, ShapeLLM-13B, achieving 55.0%, 50.92%, and 42.7% on the
  classification, captioning, and VQA tasks, respectively. Our results
  demonstrate that the encoder-free architecture is highly promising for
  replacing encoder-based architectures in the field of 3D understanding. The
  code is released at https://github.com/Ivan-Tang-3D/ENEL
paper_authors._id: 67aeec91b1bbfb68824df5d1
paper_authors.name: Yiwen Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:00:57.216Z
paper_authors._id: 6552f1ad5d55ccb20e9142a0
paper_authors.avatarUrl: /avatars/0e3e80cba64b5ae0bc5638694ac33dbf.svg
paper_authors.fullname: Ivan Tang
paper_authors.type: user
paper_authors.user: IvanTang
paper_authors._id: 67aeec91b1bbfb68824df5d2
paper_authors.name: Zoey Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:02:40.282Z
paper_authors._id: 642a8302d651bae3c11b72b1
paper_authors.avatarUrl: /avatars/4d2d422613e274d80482fed9a7d3f785.svg
paper_authors.fullname: Zoey Guo
paper_authors.type: user
paper_authors.user: Purple1288
paper_authors._id: 67aeec91b1bbfb68824df5d3
paper_authors.name: Zhuhao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:02:47.566Z
paper_authors._id: 672489e32fd598f719be33ba
paper_authors.avatarUrl: /avatars/63a6edf7bf38957e029fa52c1a7f9061.svg
paper_authors.fullname: Zhuhao Wang
paper_authors.type: user
paper_authors.user: zhuhaow
paper_authors._id: 67aeec91b1bbfb68824df5d4
paper_authors.name: Ray Zhang
paper_authors._id: 67aeec91b1bbfb68824df5d5
paper_authors.name: Qizhi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:03:12.952Z
paper_authors._id: 6535045a910b844786a6642f
paper_authors.avatarUrl: /avatars/37a94864a7a348151837b421ea6d77e3.svg
paper_authors.fullname: Qizhi Chen
paper_authors.type: user
paper_authors.user: Tavish9
paper_authors._id: 67aeec91b1bbfb68824df5d6
paper_authors.name: Junli Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:25:04.055Z
paper_authors._id: 656850a60e4b5ff9d552e825
paper_authors.avatarUrl: /avatars/450281494d8e683a60799bb8d4ddfff1.svg
paper_authors.fullname: Junli Liu
paper_authors.type: user
paper_authors.user: Ideallll
paper_authors._id: 67aeec91b1bbfb68824df5d7
paper_authors.name: Delin Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-14T08:00:55.263Z
paper_authors._id: 64daecec888b7e9c400f59b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64daecec888b7e9c400f59b5/f4pfOfWk6jYJX-Nf2-qHn.png
paper_authors.fullname: Delin Qu
paper_authors.type: user
paper_authors.user: delinqu
paper_authors._id: 67aeec91b1bbfb68824df5d8
paper_authors.name: Zhigang Wang
paper_authors._id: 67aeec91b1bbfb68824df5d9
paper_authors.name: Dong Wang
paper_authors._id: 67aeec91b1bbfb68824df5da
paper_authors.name: Xuelong Li
paper_authors._id: 67aeec91b1bbfb68824df5db
paper_authors.name: Bin Zhao
paper_title: Exploring the Potential of Encoder-free Architectures in 3D LMMs
paper_summary: Encoder-free architectures have been preliminarily explored in the 2D visual
  domain, yet it remains an open question whether they can be effectively applied
  to 3D understanding scenarios. In this paper, we present the first
  comprehensive investigation into the potential of encoder-free architectures to
  overcome the challenges of encoder-based 3D Large Multimodal Models (LMMs).
  These challenges include the failure to adapt to varying point cloud
  resolutions and the point features from the encoder not meeting the semantic
  needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to
  remove the encoder and enable the LLM to assume the role of the 3D encoder: 1)
  We propose the LLM-embedded Semantic Encoding strategy in the pre-training
  stage, exploring the effects of various point cloud self-supervised losses. And
  we present the Hybrid Semantic Loss to extract high-level semantics. 2) We
  introduce the Hierarchical Geometry Aggregation strategy in the instruction
  tuning stage. This incorporates inductive bias into the LLM early layers to
  focus on the local details of the point clouds. To the end, we present the
  first Encoder-free 3D LMM, ENEL. Our 7B model rivals the current
  state-of-the-art model, ShapeLLM-13B, achieving 55.0%, 50.92%, and 42.7% on the
  classification, captioning, and VQA tasks, respectively. Our results
  demonstrate that the encoder-free architecture is highly promising for
  replacing encoder-based architectures in the field of 3D understanding. The
  code is released at https://github.com/Ivan-Tang-3D/ENEL
paper_ai_keywords: encoder-free architectures 3D Large Multimodal Models (LMMs) point cloud resolutions point features Large Language Models (LLMs) LLM-embedded Semantic Encoding point cloud self-supervised losses Hybrid Semantic Loss Hierarchical Geometry Aggregation inductive bias ENEL ShapeLLM-13B classification captioning VQA tasks