Attention, Please! Revisiting Attentive Probing for Masked Image
  Modeling
As fine-tuning (FT) becomes increasingly impractical at scale, probing is
emerging as the preferred evaluation protocol for self-supervised learning
(SSL). Yet, the standard linear probing (LP) fails to adequately reflect the
potential of models trained with Masked Image Modeling (MIM), due to the
distributed nature of patch tokens. This motivates the need for attentive
probing, an alternative that uses attention to selectively aggregate
patch-level features. Despite its growing adoption, attentive probing remains
under-explored, with existing methods suffering from excessive parameterization
and poor computational efficiency.
  In this work, we revisit attentive probing through the lens of the
accuracy-efficiency trade-off. We conduct a systematic study of existing
methods, analyzing their mechanisms and benchmarking their performance. We
introduce efficient probing (EP), a multi-query cross-attention mechanism that
eliminates redundant projections, reduces the number of trainable parameters,
and achieves up to a 10times speed-up over conventional multi-head
attention. Despite its simplicity, EP outperforms LP and prior attentive
probing approaches across seven benchmarks, generalizes well beyond MIM to
diverse pre-training paradigms, produces interpretable attention maps, and
achieves strong gains in low-shot and layer-wise settings. Code available at
https://github.com/billpsomas/efficient-probing.
684bfd5d3b733ba3336871f1
Bill Psomas
claimed_verified
2025-06-13T14:59:18.359Z
626a9b5205fe1cb65720e00e
https://cdn-avatars.huggingface.co/v1/production/uploads/626a9b5205fe1cb65720e00e/hyWcWn_8jVZsu1Yc5Z0R8.png
Bill Psomas
user
billpsomas
684bfd5d3b733ba3336871f2
Dionysis Christopoulos
claimed_verified
2025-06-20T12:20:17.756Z
684c0370a68abf6c4698459d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/SO3fyL030PtRb6g0ZBa5Y.png
Dionysis Christopoulos
user
DionysisChr
684bfd5d3b733ba3336871f3
Eirini Baltzi
claimed_verified
2025-06-24T09:33:09.239Z
684c0118bd8ce5496375d16f
https://cdn-avatars.huggingface.co/v1/production/uploads/684c0118bd8ce5496375d16f/hnS1d9UpD1eLr9OGdUZe3.jpeg
Eirini Baltzi
user
EiriniBaltzi
684bfd5d3b733ba3336871f4
Ioannis Kakogeorgiou
claimed_verified
2025-07-09T08:53:21.883Z
661ba524bd9243bf7e598355
https://cdn-avatars.huggingface.co/v1/production/uploads/661ba524bd9243bf7e598355/i77yD4XgJn2vUbn_mIsT8.jpeg
Ioannis Kakogeorgiou
user
gkakogeorgiou
684bfd5d3b733ba3336871f5
Tilemachos Aravanis
684bfd5d3b733ba3336871f6
Nikos Komodakis
684bfd5d3b733ba3336871f7
Konstantinos Karantzalos
684bfd5d3b733ba3336871f8
Yannis Avrithis
684bfd5d3b733ba3336871f9
Giorgos Tolias
Attention, Please! Revisiting Attentive Probing for Masked Image
  Modeling
As fine-tuning (FT) becomes increasingly impractical at scale, probing is
emerging as the preferred evaluation protocol for self-supervised learning
(SSL). Yet, the standard linear probing (LP) fails to adequately reflect the
potential of models trained with Masked Image Modeling (MIM), due to the
distributed nature of patch tokens. This motivates the need for attentive
probing, an alternative that uses attention to selectively aggregate
patch-level features. Despite its growing adoption, attentive probing remains
under-explored, with existing methods suffering from excessive parameterization
and poor computational efficiency.
  In this work, we revisit attentive probing through the lens of the
accuracy-efficiency trade-off. We conduct a systematic study of existing
methods, analyzing their mechanisms and benchmarking their performance. We
introduce efficient probing (EP), a multi-query cross-attention mechanism that
eliminates redundant projections, reduces the number of trainable parameters,
and achieves up to a 10times speed-up over conventional multi-head
attention. Despite its simplicity, EP outperforms LP and prior attentive
probing approaches across seven benchmarks, generalizes well beyond MIM to
diverse pre-training paradigms, produces interpretable attention maps, and
achieves strong gains in low-shot and layer-wise settings. Code available at
https://github.com/billpsomas/efficient-probing.
Efficient probing using a multi-query cross-attention mechanism enhances performance in self-supervised learning while reducing parameterization and computational cost compared to traditional methods.
fine-tuning self-supervised learning Masked Image Modeling linear probing attentive probing patch tokens attention patch-level features multi-query cross-attention parameterization computational efficiency accuracy-efficiency trade-off benchmarks multi-head attention low-shot layer-wise