  Since the introduction of the Vision Transformer (ViT), researchers have
sought to make ViTs more efficient by removing redundant information in the
processed tokens. While different methods have been explored to achieve this
goal, we still lack understanding of the resulting reduction patterns and how
those patterns differ across token reduction methods and datasets. To close
this gap, we set out to understand the reduction patterns of 10 different token
reduction methods using four image classification datasets. By systematically
comparing these methods on the different classification tasks, we find that the
Top-K pruning method is a surprisingly strong baseline. Through in-depth
analysis of the different methods, we determine that: the reduction patterns
are generally not consistent when varying the capacity of the backbone model,
the reduction patterns of pruning-based methods significantly differ from fixed
radial patterns, and the reduction patterns of pruning-based methods are
correlated across classification datasets. Finally we report that the
similarity of reduction patterns is a moderate-to-strong proxy for model
performance. Project page at https://vap.aau.dk/tokens.

Computer Vision and Pattern Recognition