title: Pre-training Small Base LMs with Fewer Tokens
summary: We study the effectiveness of a simple approach to develop a small base
  language model (LM) starting from an existing large base LM: first inherit a
  few transformer blocks from the larger LM, and then train this smaller model on
  a very small subset (0.1\%) of the raw pretraining data of the larger model. We
  call our simple recipe Inheritune and first demonstrate it for building a small
  base LM with 1.5B parameters using 1B tokens (and a starting few layers of
  larger LM of 3B parameters); we do this using a single A6000 GPU for less than
  half a day. Across 9 diverse evaluation datasets as well as the MMLU benchmark,
  the resulting model compares favorably to publicly available base models of
  1B-2B size, some of which have been trained using 50-1000 times more tokens.
    We investigate Inheritune in a slightly different setting where we train
  small LMs utilizing larger LMs and their full pre-training dataset. Here we
  show that smaller LMs trained utilizing some of the layers of GPT2-medium
  (355M) and GPT-2-large (770M) can effectively match the val loss of their
  bigger counterparts when trained from scratch for the same number of training
  steps on OpenWebText dataset with 9B tokens. We analyze our recipe with
  extensive experiments and demonstrate it efficacy on diverse settings. Our code
  is available at https://github.com/sanyalsunny111/LLM-Inheritune.
paper_authors._id: 661c85fd3557013b631d5dbe
paper_authors.name: Sunny Sanyal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:14:18.376Z
paper_authors._id: 631909e582e7eec0eabca95e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662585187921-noauth.png
paper_authors.fullname: Sunny Sanyal
paper_authors.type: user
paper_authors.user: Sunny111
paper_authors._id: 661c85fd3557013b631d5dbf
paper_authors.name: Sujay Sanghavi
paper_authors._id: 661c85fd3557013b631d5dc0
paper_authors.name: Alexandros G. Dimakis
paper_title: Pre-training Small Base LMs with Fewer Tokens
paper_summary: We study the effectiveness of a simple approach to develop a small base
  language model (LM) starting from an existing large base LM: first inherit a
  few transformer blocks from the larger LM, and then train this smaller model on
  a very small subset (0.1\%) of the raw pretraining data of the larger model. We
  call our simple recipe Inheritune and first demonstrate it for building a small
  base LM with 1.5B parameters using 1B tokens (and a starting few layers of
  larger LM of 3B parameters); we do this using a single A6000 GPU for less than
  half a day. Across 9 diverse evaluation datasets as well as the MMLU benchmark,
  the resulting model compares favorably to publicly available base models of
  1B-2B size, some of which have been trained using 50-1000 times more tokens.
    We investigate Inheritune in a slightly different setting where we train
  small LMs utilizing larger LMs and their full pre-training dataset. Here we
  show that smaller LMs trained utilizing some of the layers of GPT2-medium
  (355M) and GPT-2-large (770M) can effectively match the val loss of their
  bigger counterparts when trained from scratch for the same number of training
  steps on OpenWebText dataset with 9B tokens. We analyze our recipe with
  extensive experiments and demonstrate it efficacy on diverse settings. Our code
  is available at https://github.com/sanyalsunny111/LLM-Inheritune.
paper_ai_keywords: transformer blocks base language model (LM) pretraining data Inheritune evaluation datasets MMLU benchmark GPT2-medium GPT-2-large OpenWebText dataset val loss training steps