REOrdering Patches Improves Vision Models
Sequence models such as transformers require inputs to be represented as
one-dimensional sequences. In vision, this typically involves flattening images
using a fixed row-major (raster-scan) order. While full self-attention is
permutation-equivariant, modern long-sequence transformers increasingly rely on
architectural approximations that break this invariance and introduce
sensitivity to patch ordering. We show that patch order significantly affects
model performance in such settings, with simple alternatives like column-major
or Hilbert curves yielding notable accuracy shifts. Motivated by this, we
propose REOrder, a two-stage framework for discovering task-optimal patch
orderings. First, we derive an information-theoretic prior by evaluating the
compressibility of various patch sequences. Then, we learn a policy over
permutations by optimizing a Plackett-Luce policy using REINFORCE. This
approach enables efficient learning in a combinatorial permutation space.
REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to
3.01% and Functional Map of the World by 13.35%.
683907321bf7a7a94309d26b
Declan Kutscher
claimed_verified
2025-05-30T06:54:30.099Z
665bed48cd4115f092a0f750
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Muvzf256flNUAa8mBwkov.png
Declan Kutscher
user
d3tk
683907321bf7a7a94309d26c
David M. Chan
683907321bf7a7a94309d26d
Yutong Bai
683907321bf7a7a94309d26e
Trevor Darrell
683907321bf7a7a94309d26f
Ritwik Gupta
claimed_verified
2025-05-30T06:54:28.074Z
630489e5dae2eb7d083e78b1
https://cdn-avatars.huggingface.co/v1/production/uploads/630489e5dae2eb7d083e78b1/RKjwiquU1JzaHoV4op42F.jpeg
Ritwik Gupta
user
RitwikGupta
REOrdering Patches Improves Vision Models
Sequence models such as transformers require inputs to be represented as
one-dimensional sequences. In vision, this typically involves flattening images
using a fixed row-major (raster-scan) order. While full self-attention is
permutation-equivariant, modern long-sequence transformers increasingly rely on
architectural approximations that break this invariance and introduce
sensitivity to patch ordering. We show that patch order significantly affects
model performance in such settings, with simple alternatives like column-major
or Hilbert curves yielding notable accuracy shifts. Motivated by this, we
propose REOrder, a two-stage framework for discovering task-optimal patch
orderings. First, we derive an information-theoretic prior by evaluating the
compressibility of various patch sequences. Then, we learn a policy over
permutations by optimizing a Plackett-Luce policy using REINFORCE. This
approach enables efficient learning in a combinatorial permutation space.
REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to
3.01% and Functional Map of the World by 13.35%.
REOrder discovers task-optimal patch orderings for long-sequence transformers, significantly improving accuracy over traditional ordering methods.
sequence models transformers self-attention permutation-equivariant patch ordering column-major Hilbert curves information-theoretic prior Plackett-Luce policy REINFORCE ImageNet-1K Functional Map of the World