TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly
<|reference_start|>TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly: Inferring step-wise actions to assemble 3D objects with primitive bricks from images is a challenging task due to complex constraints and the vast number of possible combinations. Recent studies have demonstrated promising results on sequential LEGO brick assembly through the utilization of LEGO-Graph modeling to predict sequential actions. However, existing approaches are class-specific and require significant computational and 3D annotation resources. In this work, we first propose a computationally efficient breadth-first search (BFS) LEGO-Tree structure to model the sequential assembly actions by considering connections between consecutive layers. Based on the LEGO-Tree structure, we then design a class-agnostic tree-transformer framework to predict the sequential assembly actions from the input multi-view images. A major challenge of the sequential brick assembly task is that the step-wise action labels are costly and tedious to obtain in practice. We mitigate this problem by leveraging synthetic-to-real transfer learning. Specifically, our model is first pre-trained on synthetic data with full supervision from the available action labels. We then circumvent the requirement for action labels in the real data by proposing an action-to-silhouette projection that replaces action labels with input image silhouettes for self-supervision. Without any annotation on the real data, our model outperforms existing methods with 3D supervision by 7.8% and 11.3% in mIoU on the MNIST and ModelNet Construction datasets, respectively.<|reference_end|>
@article{guo2024treesba:,
    title={TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly},
    author={Mengqi Guo, Chen Li, Yuyang Zhao, Gim Hee Lee},
    journal={arXiv preprint arXiv:2407.15648},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15648},
    primaryClass={cs.CV}
}