--- '[object Object]': null language: - en license: other license_name: autodesk-non-commercial-3d-generative-v1.0 license_link: LICENSE.md tags: - make-a-shape - mv-to-3d pipeline_tag: image-to-3d --- --- # Model Card for Make-A-Shape Multi-View to 3D Model This model is part of the Make-A-Shape paper, capable of generating high-quality 3D shapes from multi-view images with intricate geometric details, realistic structures, and complex topologies. ## Model Details ### Model Description Make-A-Shape is a novel 3D generative framework trained on an extensive dataset of over 10 million publicly-available 3D shapes. The multi-view to 3D model is one of the conditional generation models in this framework. It can efficiently generate a wide range of high-quality 3D shapes from four view-specific images as inputs. The model uses a wavelet-tree representation and adaptive training strategy to achieve superior performance in terms of geometric detail and structural plausibility. - **Developed by:** Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, Chi-Wing Fu - **Model type:** 3D Generative Model - **License:** Autodesk Non-Commercial (3D Generative) v1.0 For more information please look at the [Project](https://www.research.autodesk.com/publications/generative-ai-make-a-shape/) [Page](https://edward1997104.github.io/make-a-shape/) and [the ICML paper](https://proceedings.mlr.press/v235/hui24a.html). ### Model Sources - **Repository:** [https://github.com/AutodeskAILab/Make-a-Shape](https://github.com/AutodeskAILab/Make-a-Shape) - **Paper:** [ArXiv:2401.11067](https://arxiv.org/abs/2401.11067), [ICML - Make-A-Shape: a Ten-Million-scale 3D Shape Model](https://proceedings.mlr.press/v235/hui24a.html) - **Demo:** [Google Colab](https://colab.research.google.com/drive/1XIoeanLjXIDdLow6qxY7cAZ6YZpqY40d?usp=sharing) ## Uses ### Direct Use This model is specifically developed for the sole purpose of scientific research and development. This model is released by Autodesk and intended for academic and research purposes only for the theoretical exploration and demonstration of the Make-a-Shape 3D generative framework. Please see [here](https://github.com/AutodeskAILab/Make-a-Shape?tab=readme-ov-file#multi-view-to-3d) for inferencing instructions. ### Out-of-Scope Use The model should not be used for: - Commercial purposes - Creation of load-bearing physical objects the failure of which could cause property damage or personal injury - Any usage not in compliance with the [license](https://huggingface.co/ADSKAILab/Make-A-Shape-multi-view-20m/blob/main/LICENSE.md), in particular, the "Acceptable Use" section. ## Bias, Risks, and Limitations ### Bias - The model may inherit biases present in the publicly-available training datasets, which could lead to uneven representation of certain object types or styles. - The model's performance may degrade for object categories or styles that are underrepresented in the training data. ### Risks and Limitations - The quality of the generated 3D output may be impacted by the quality and clarity of the input image. - The model may occasionally generate implausible shapes, especially when the input image is ambiguous or of low quality. Even theoretically plausible shapes should not be relied upon for real-world structural soundness. ## How to Get Started with the Model Please refer to the instructions [here](https://github.com/AutodeskAILab/Make-a-Shape?tab=readme-ov-file#multi-view-to-3d). ## Training Details ### Training Data The model was trained on a dataset of over 10 million 3D shapes aggregated from 18 different publicly-available sub-datasets, including ModelNet, ShapeNet, SMPL, Thingi10K, SMAL, COMA, House3D, ABC, Fusion 360, 3D-FUTURE, BuildingNet, DeformingThings4D, FG3D, Toys4K, ABO, Infinigen, Objaverse, and two subsets of ObjaverseXL (Thingiverse and GitHub). ### Training Procedure #### Preprocessing Each 3D shape in the dataset was converted into a truncated signed distance function (TSDF) with a resolution of 256³. The TSDF was then decomposed using a discrete wavelet transform to create the wavelet-tree representation used by the model. #### Training Hyperparameters - **Training regime:** Please refer to the paper. #### Speeds, Sizes, Times - The model was trained on 48 × A10G GPUs for about 20 days, amounting to around 23,000 GPU hours. - The model can generate shapes within two seconds for most conditions. ## Evaluation ### Testing Data, Factors & Metrics #### Testing Data The model was evaluated on a test set consisting of 2% of the shapes from each sub-dataset in the training data, as well as on the entire Google Scanned Objects (GSO) dataset, which was not part of the training data. #### Factors The evaluation considered various factors such as the quality of generated shapes, the ability to capture fine details and complex structures, and the model's performance across different object categories. #### Metrics The model was evaluated using the following metrics: - Intersection over Union (IoU) - Light Field Distance (LFD) - Chamfer Distance (CD) ### Results The multi-view to 3D model achieved the following results on the "Our Val" dataset: - LFD: 2217.25 - IoU: 0.6707 - CD: 0.00350 On the GSO dataset: - LFD: 1890.85 - IoU: 0.7460 - CD: 0.00337 ## Technical Specifications ### Model Architecture and Objective The model uses a U-ViT architecture with learnable skip-connections between the convolution and deconvolution blocks. It employs a wavelet-tree representation and a subband adaptive training strategy to effectively capture both coarse and fine details of 3D shapes. ### Compute Infrastructure #### Hardware The model was trained on 48 × A10G GPUs. ## Citation **BibTeX:** ```latex @InProceedings{pmlr-v235-hui24a, title = {Make-A-Shape: a Ten-Million-scale 3{D} Shape Model}, author = {Hui, Ka-Hei and Sanghi, Aditya and Rampini, Arianna and Rahimi Malekshan, Kamal and Liu, Zhengzhe and Shayani, Hooman and Fu, Chi-Wing}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {20660--20681}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hui24a/hui24a.pdf}, url = {https://proceedings.mlr.press/v235/hui24a.html}, } ```