pix2gestalt: Amodal Segmentation by Synthesizing Wholes
We introduce pix2gestalt, a framework for zero-shot amodal segmentation,
which learns to estimate the shape and appearance of whole objects that are
only partially visible behind occlusions. By capitalizing on large-scale
diffusion models and transferring their representations to this task, we learn
a conditional diffusion model for reconstructing whole objects in challenging
zero-shot cases, including examples that break natural and physical priors,
such as art. As training data, we use a synthetically curated dataset
containing occluded objects paired with their whole counterparts. Experiments
show that our approach outperforms supervised baselines on established
benchmarks. Our model can furthermore be used to significantly improve the
performance of existing object recognition and 3D reconstruction methods in the
presence of occlusions.
65b33a5a2be9660f0b425af0
Ege Ozguroglu
claimed_verified
2024-01-26T08:23:36.337Z
65b34a907cd0069ad64b5daa
/avatars/efc321bd987ad907c53c5a07e81a6aea.svg
Ege Ozguroglu
user
egeozguroglu
65b33a5a2be9660f0b425af1
Ruoshi Liu
admin_assigned
2024-01-26T09:48:03.265Z
63408322ea76a1e0b47b50e8
/avatars/8e2453313006fdf70b99bc7d29c738b1.svg
Ruoshi Liu
user
rliu
65b33a5a2be9660f0b425af2
Dídac Surís
admin_assigned
2024-01-26T09:48:15.447Z
6303e8314ec2dfa82a56ed8b
/avatars/91a80a05396ad575ca50571035b2bc09.svg
Didac Suris
user
didacsuris
65b33a5a2be9660f0b425af3
Dian Chen
admin_assigned
2024-01-26T09:48:37.439Z
63a240fbe36f2e4d5b1b393c
/avatars/2cf95ab486d3473cf0184b3844a21987.svg
Dian Chen
user
dianchen
65b33a5a2be9660f0b425af4
Achal Dave
admin_assigned
2024-01-26T09:48:52.905Z
6488e29b78570cedad71eb2d
/avatars/0d8bb52c1e2be8d6aa7c732cf92fc5a9.svg
Achal Dave
user
achal-tri
65b33a5a2be9660f0b425af5
Pavel Tokmakov
admin_assigned
2024-01-26T09:48:58.780Z
65aeab2201b6f61e24323339
/avatars/2c119173c0871f59df108f9bb4471954.svg
Pavel Tokmakov
user
ptokmakov
65b33a5a2be9660f0b425af6
Carl Vondrick
admin_assigned
2024-01-26T09:49:05.228Z
63a4bfcbcde2b28f82a2c50c
/avatars/9d978a6f0caf6ad92ec4138a84a39e53.svg
Carl Vondrick
user
cvondrick
pix2gestalt: Amodal Segmentation by Synthesizing Wholes
We introduce pix2gestalt, a framework for zero-shot amodal segmentation,
which learns to estimate the shape and appearance of whole objects that are
only partially visible behind occlusions. By capitalizing on large-scale
diffusion models and transferring their representations to this task, we learn
a conditional diffusion model for reconstructing whole objects in challenging
zero-shot cases, including examples that break natural and physical priors,
such as art. As training data, we use a synthetically curated dataset
containing occluded objects paired with their whole counterparts. Experiments
show that our approach outperforms supervised baselines on established
benchmarks. Our model can furthermore be used to significantly improve the
performance of existing object recognition and 3D reconstruction methods in the
presence of occlusions.
Pix2gestalt uses conditional diffusion models, trained on synthetic occluded-object datasets, to achieve zero-shot amodal segmentation better than supervised approaches and enhances object recognition and 3D reconstruction in occlusions.
pix2gestalt zero-shot amodal segmentation diffusion models conditional diffusion model occlusions synthetic dataset object recognition 3D reconstruction