title: The Impact of Depth and Width on Transformer Language Model
    Generalization
summary: To process novel sentences, language models (LMs) must generalize
  compositionally -- combine familiar elements in new ways. What aspects of a
  model's structure promote compositional generalization? Focusing on
  transformers, we test the hypothesis, motivated by recent theoretical and
  empirical work, that transformers generalize more compositionally when they are
  deeper (have more layers). Because simply adding layers increases the total
  number of parameters, confounding depth and size, we construct three classes of
  models which trade off depth for width such that the total number of parameters
  is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs
  and fine-tune them on tasks that test for compositional generalization. We
  report three main conclusions: (1) after fine-tuning, deeper models generalize
  better out-of-distribution than shallower models do, but the relative benefit
  of additional layers diminishes rapidly; (2) within each family, deeper models
  show better language modeling performance, but returns are similarly
  diminishing; (3) the benefits of depth for compositional generalization cannot
  be attributed solely to better performance on language modeling or on
  in-distribution data.
paper_authors._id: 6541b9a2c1737a4320dc0ff9
paper_authors.name: Jackson Petty
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-21T19:34:08.414Z
paper_authors._id: 655cbcde95959f36af636d9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655cbcde95959f36af636d9a/wV-pwUpUeDOjX9iUNa9Zq.jpeg
paper_authors.fullname: Jackson Petty
paper_authors.type: user
paper_authors.user: jowenpetty
paper_authors._id: 6541b9a2c1737a4320dc0ffa
paper_authors.name: Sjoerd van Steenkiste
paper_authors._id: 6541b9a2c1737a4320dc0ffb
paper_authors.name: Ishita Dasgupta
paper_authors._id: 6541b9a2c1737a4320dc0ffc
paper_authors.name: Fei Sha
paper_authors._id: 6541b9a2c1737a4320dc0ffd
paper_authors.name: Dan Garrette
paper_authors._id: 6541b9a2c1737a4320dc0ffe
paper_authors.name: Tal Linzen
paper_title: The Impact of Depth and Width on Transformer Language Model
    Generalization
paper_summary: To process novel sentences, language models (LMs) must generalize
  compositionally -- combine familiar elements in new ways. What aspects of a
  model's structure promote compositional generalization? Focusing on
  transformers, we test the hypothesis, motivated by recent theoretical and
  empirical work, that transformers generalize more compositionally when they are
  deeper (have more layers). Because simply adding layers increases the total
  number of parameters, confounding depth and size, we construct three classes of
  models which trade off depth for width such that the total number of parameters
  is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs
  and fine-tune them on tasks that test for compositional generalization. We
  report three main conclusions: (1) after fine-tuning, deeper models generalize
  better out-of-distribution than shallower models do, but the relative benefit
  of additional layers diminishes rapidly; (2) within each family, deeper models
  show better language modeling performance, but returns are similarly
  diminishing; (3) the benefits of depth for compositional generalization cannot
  be attributed solely to better performance on language modeling or on
  in-distribution data.
paper_ai_keywords: transformers compositional generalization pretrain fine-tune language modeling in-distribution data out-of-distribution