NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and
  Repulsive UDF
Remarkable progress has been made in 3D reconstruction from single-view RGB-D
inputs. MCC is the current state-of-the-art method in this field, which
achieves unprecedented success by combining vision Transformers with
large-scale training. However, we identified two key limitations of MCC: 1) The
Transformer decoder is inefficient in handling large number of query points; 2)
The 3D representation struggles to recover high-fidelity details. In this
paper, we propose a new approach called NU-MCC that addresses these
limitations. NU-MCC includes two key innovations: a Neighborhood decoder and a
Repulsive Unsigned Distance Function (Repulsive UDF). First, our Neighborhood
decoder introduces center points as an efficient proxy of input visual
features, allowing each query point to only attend to a small neighborhood.
This design not only results in much faster inference speed but also enables
the exploitation of finer-scale visual features for improved recovery of 3D
textures. Second, our Repulsive UDF is a novel alternative to the occupancy
field used in MCC, significantly improving the quality of 3D object
reconstruction. Compared to standard UDFs that suffer from holes in results,
our proposed Repulsive UDF can achieve more complete surface reconstruction.
Experimental results demonstrate that NU-MCC is able to learn a strong 3D
representation, significantly advancing the state of the art in single-view 3D
reconstruction. Particularly, it outperforms MCC by 9.7% in terms of the
F1-score on the CO3D-v2 dataset with more than 5x faster running speed.
64b75b77da8017900e776d36
Stefan Lionar
claimed_verified
2023-07-19T14:39:15.873Z
63bfb9a54ed1b8fd6883524c
/avatars/5462264a8cf58cb15fcc5194336d45ac.svg
Stefan Lionar
user
slionar
64b75b77da8017900e776d37
Xiangyu Xu
claimed_verified
2023-07-19T14:39:12.284Z
634e60454677a5891c0902f4
/avatars/4dc143719afe7686e05b7f2c2c5c1871.svg
Xiangyu Xu
user
xjcvcvxj
64b75b77da8017900e776d38
Min Lin
claimed_verified
2024-01-22T12:01:11.551Z
6565d5fd656151d58b1d9afb
/avatars/0c416ea74347c6b5bd27f49aec309c48.svg
Min Lin
user
mavenlin
64b75b77da8017900e776d39
Gim Hee Lee
claimed_verified
2023-07-24T07:44:04.007Z
64bc7568b7375f6b84503ce2
/avatars/d970c9990d9d84ac51ff225c1b26e304.svg
Gim Hee Lee
user
gimhee-lee
NU-MCC: Multiview Compressive Coding with Neighborhood Decoder and
  Repulsive UDF
Remarkable progress has been made in 3D reconstruction from single-view RGB-D
inputs. MCC is the current state-of-the-art method in this field, which
achieves unprecedented success by combining vision Transformers with
large-scale training. However, we identified two key limitations of MCC: 1) The
Transformer decoder is inefficient in handling large number of query points; 2)
The 3D representation struggles to recover high-fidelity details. In this
paper, we propose a new approach called NU-MCC that addresses these
limitations. NU-MCC includes two key innovations: a Neighborhood decoder and a
Repulsive Unsigned Distance Function (Repulsive UDF). First, our Neighborhood
decoder introduces center points as an efficient proxy of input visual
features, allowing each query point to only attend to a small neighborhood.
This design not only results in much faster inference speed but also enables
the exploitation of finer-scale visual features for improved recovery of 3D
textures. Second, our Repulsive UDF is a novel alternative to the occupancy
field used in MCC, significantly improving the quality of 3D object
reconstruction. Compared to standard UDFs that suffer from holes in results,
our proposed Repulsive UDF can achieve more complete surface reconstruction.
Experimental results demonstrate that NU-MCC is able to learn a strong 3D
representation, significantly advancing the state of the art in single-view 3D
reconstruction. Particularly, it outperforms MCC by 9.7% in terms of the
F1-score on the CO3D-v2 dataset with more than 5x faster running speed.
A new method, NU-MCC, addresses limitations in 3D reconstruction by using a Neighborhood decoder and a Repulsive Unsigned Distance Function, improving inference speed and reconstruction quality compared to the current state-of-the-art method.
vision Transformers Transformer decoder Neighborhood decoder Repulsive Unsigned Distance Function occupancy field 3D reconstruction CO3D-v2 dataset