title: Transformer Layers as Painters
summary: Despite their nearly universal adoption for large language models, the
  internal workings of transformers are not well understood. We aim to better
  understand the impact of removing or reorganizing information throughout the
  layers of a pretrained transformer. Such an understanding could both yield
  better usage of existing models as well as to make architectural improvements
  to produce new variants. We present a series of empirical studies on frozen
  models that show that the lower and final layers of pretrained transformers
  differ from middle layers, but that middle layers have a surprising amount of
  uniformity. We further show that some classes of problems have robustness to
  skipping layers, running the layers in an order different from how they were
  trained, or running the layers in parallel. Our observations suggest that even
  frozen pretrained models may gracefully trade accuracy for latency by skipping
  layers or running layers in parallel.
paper_authors._id: 669489afcab4aab691e57e1a
paper_authors.name: Qi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-05T09:02:32.335Z
paper_authors._id: 644b983f0fbe4830f192c4f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b983f0fbe4830f192c4f5/4eX4OS0gnXGqLu3eaMml_.png
paper_authors.fullname: Qi Sun
paper_authors.type: user
paper_authors.user: lfsm
paper_authors._id: 669489afcab4aab691e57e1b
paper_authors.name: Marc Pickett
paper_authors._id: 669489afcab4aab691e57e1c
paper_authors.name: Aakash Kumar Nain
paper_authors._id: 669489afcab4aab691e57e1d
paper_authors.name: Llion Jones
paper_title: Transformer Layers as Painters
paper_summary: Despite their nearly universal adoption for large language models, the
  internal workings of transformers are not well understood. We aim to better
  understand the impact of removing or reorganizing information throughout the
  layers of a pretrained transformer. Such an understanding could both yield
  better usage of existing models as well as to make architectural improvements
  to produce new variants. We present a series of empirical studies on frozen
  models that show that the lower and final layers of pretrained transformers
  differ from middle layers, but that middle layers have a surprising amount of
  uniformity. We further show that some classes of problems have robustness to
  skipping layers, running the layers in an order different from how they were
  trained, or running the layers in parallel. Our observations suggest that even
  frozen pretrained models may gracefully trade accuracy for latency by skipping
  layers or running layers in parallel.
paper_ai_keywords: transformers pretrained transformers frozen models lower layers final layers middle layers latency skipping layers parallelization robustness accuracy