Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding
mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
for masked patch reconstruction in MAE into self-attention and cross-attention.
Our investigations suggest that self-attention between mask patches is not
essential for learning good representations. To this end, we propose a novel
pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
CrossMAE's decoder leverages only cross-attention between masked and visible
tokens, with no degradation in downstream performance. This design also enables
decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
each decoder block can now leverage different encoder features, resulting in
improved representation learning. CrossMAE matches MAE in performance with 2.5
to 3.7times less decoding compute. It also surpasses MAE on ImageNet
classification and COCO instance segmentation under the same compute. Code and
models: https://crossmae.github.io
65b31a603a41095a56ae0fc5
Letian Fu
admin_assigned
2024-01-26T08:22:58.402Z
64908d6b0c18343a09361b93
/avatars/2fa44344fe760f68f19de775b76f45b8.svg
Max (Letian) Fu
user
mlfu7
65b31a603a41095a56ae0fc6
Long Lian
admin_assigned
2024-01-26T08:25:25.404Z
63797c273f575acc2f6893c0
/avatars/32d7a6a8881c8c4d80a097b732ed24b6.svg
Long(Tony) Lian
user
longlian
65b31a603a41095a56ae0fc7
Renhao Wang
65b31a603a41095a56ae0fc8
Baifeng Shi
admin_assigned
2024-01-26T08:26:01.815Z
649004218f7cbbc94c782db6
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AdgLVfAIpWlug4jXTaEK-.jpeg
Baifeng Shi
user
bfshi
65b31a603a41095a56ae0fc9
Xudong Wang
admin_assigned
2024-01-26T08:26:35.984Z
63d6f76f44f1d8fbe58fbd9d
/avatars/184f248a090b543e576a97b7f46ab3ef.svg
Xudong Wang
user
xudongw
65b31a603a41095a56ae0fca
Adam Yala
admin_assigned
2024-01-26T08:26:55.544Z
6333a9195a032dcd095dda13
https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
Adam Yala
user
yala
65b31a603a41095a56ae0fcb
Trevor Darrell
admin_assigned
2024-01-26T08:27:03.308Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
65b31a603a41095a56ae0fcc
Alexei A. Efros
65b31a603a41095a56ae0fcd
Ken Goldberg
Rethinking Patch Dependence for Masked Autoencoders
In this work, we re-examine inter-patch dependencies in the decoding
mechanism of masked autoencoders (MAE). We decompose this decoding mechanism
for masked patch reconstruction in MAE into self-attention and cross-attention.
Our investigations suggest that self-attention between mask patches is not
essential for learning good representations. To this end, we propose a novel
pretraining framework: Cross-Attention Masked Autoencoders (CrossMAE).
CrossMAE's decoder leverages only cross-attention between masked and visible
tokens, with no degradation in downstream performance. This design also enables
decoding only a small subset of mask tokens, boosting efficiency. Furthermore,
each decoder block can now leverage different encoder features, resulting in
improved representation learning. CrossMAE matches MAE in performance with 2.5
to 3.7times less decoding compute. It also surpasses MAE on ImageNet
classification and COCO instance segmentation under the same compute. Code and
models: https://crossmae.github.io
By using only cross-attention for masked patch reconstruction, CrossMAE achieves comparable and sometimes superior performance to MAE with significantly reduced computational cost.
masked autoencoders self-attention cross-attention masked tokens visible tokens decoder encoder representation learning ImageNet classification COCO instance segmentation