title: REOrdering Patches Improves Vision Models
summary: Sequence models such as transformers require inputs to be represented as
  one-dimensional sequences. In vision, this typically involves flattening images
  using a fixed row-major (raster-scan) order. While full self-attention is
  permutation-equivariant, modern long-sequence transformers increasingly rely on
  architectural approximations that break this invariance and introduce
  sensitivity to patch ordering. We show that patch order significantly affects
  model performance in such settings, with simple alternatives like column-major
  or Hilbert curves yielding notable accuracy shifts. Motivated by this, we
  propose REOrder, a two-stage framework for discovering task-optimal patch
  orderings. First, we derive an information-theoretic prior by evaluating the
  compressibility of various patch sequences. Then, we learn a policy over
  permutations by optimizing a Plackett-Luce policy using REINFORCE. This
  approach enables efficient learning in a combinatorial permutation space.
  REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to
  3.01% and Functional Map of the World by 13.35%.
paper_authors._id: 683907321bf7a7a94309d26b
paper_authors.name: Declan Kutscher
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:30.099Z
paper_authors._id: 665bed48cd4115f092a0f750
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Muvzf256flNUAa8mBwkov.png
paper_authors.fullname: Declan Kutscher
paper_authors.type: user
paper_authors.user: d3tk
paper_authors._id: 683907321bf7a7a94309d26c
paper_authors.name: David M. Chan
paper_authors._id: 683907321bf7a7a94309d26d
paper_authors.name: Yutong Bai
paper_authors._id: 683907321bf7a7a94309d26e
paper_authors.name: Trevor Darrell
paper_authors._id: 683907321bf7a7a94309d26f
paper_authors.name: Ritwik Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:28.074Z
paper_authors._id: 630489e5dae2eb7d083e78b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630489e5dae2eb7d083e78b1/RKjwiquU1JzaHoV4op42F.jpeg
paper_authors.fullname: Ritwik Gupta
paper_authors.type: user
paper_authors.user: RitwikGupta
paper_title: REOrdering Patches Improves Vision Models
paper_summary: Sequence models such as transformers require inputs to be represented as
  one-dimensional sequences. In vision, this typically involves flattening images
  using a fixed row-major (raster-scan) order. While full self-attention is
  permutation-equivariant, modern long-sequence transformers increasingly rely on
  architectural approximations that break this invariance and introduce
  sensitivity to patch ordering. We show that patch order significantly affects
  model performance in such settings, with simple alternatives like column-major
  or Hilbert curves yielding notable accuracy shifts. Motivated by this, we
  propose REOrder, a two-stage framework for discovering task-optimal patch
  orderings. First, we derive an information-theoretic prior by evaluating the
  compressibility of various patch sequences. Then, we learn a policy over
  permutations by optimizing a Plackett-Luce policy using REINFORCE. This
  approach enables efficient learning in a combinatorial permutation space.
  REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to
  3.01% and Functional Map of the World by 13.35%.
paper_ai_summary: REOrder discovers task-optimal patch orderings for long-sequence transformers, significantly improving accuracy over traditional ordering methods.
paper_ai_keywords: sequence models transformers self-attention permutation-equivariant patch ordering column-major Hilbert curves information-theoretic prior Plackett-Luce policy REINFORCE ImageNet-1K Functional Map of the World