title: Pretraining Language Models for Diachronic Linguistic Change Discovery
summary: Large language models (LLMs) have shown potential as tools for scientific
  discovery. This has engendered growing interest in their use in humanistic
  disciplines, such as historical linguistics and literary studies. These fields
  often construct arguments on the basis of delineations like genre, or more
  inflexibly, time period. Although efforts have been made to restrict inference
  to specific domains via fine-tuning or model editing, we posit that the only
  true guarantee is domain-restricted pretraining -- typically, a data- and
  compute-expensive proposition.
    We show that efficient pretraining techniques can produce useful models over
  corpora too large for easy manual inspection but too small for "typical" LLM
  approaches. We employ a novel date-attribution pipeline in order to obtain a
  temporally-segmented dataset of five 10-million-word slices. We train two
  corresponding five-model batteries over these corpus segments, efficient
  pretraining and Llama3-8B parameter efficiently finetuned.
    We find that the pretrained models are faster to train than the finetuned
  baselines and that they better respect the historical divisions of our corpus.
  Emphasizing speed and precision over a-historical comprehensiveness enables a
  number of novel approaches to hypothesis discovery and testing in our target
  fields. Taking up diachronic linguistics as a testbed, we show that our method
  enables the detection of a diverse set of phenomena, including en masse lexical
  change, non-lexical (grammatical and morphological) change, and word sense
  introduction/obsolescence. We provide a ready-to-use pipeline that allows
  extension of our approach to other target fields with only minimal adaptation.
paper_authors._id: 67f7bc4c03cfd3db7e6b75e1
paper_authors.name: Elisabeth Fittschen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T15:45:39.921Z
paper_authors._id: 6706d8a8d218bb754322d731
paper_authors.avatarUrl: /avatars/807083029d5bae3cbafc05e03035b0a2.svg
paper_authors.fullname: Elisabeth Fittschen
paper_authors.type: user
paper_authors.user: efittschen
paper_authors._id: 67f7bc4c03cfd3db7e6b75e2
paper_authors.name: Sabrina Li
paper_authors._id: 67f7bc4c03cfd3db7e6b75e3
paper_authors.name: Tom Lippincott
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T13:27:35.414Z
paper_authors._id: 63418125a7582111c3f57bca
paper_authors.avatarUrl: /avatars/8128615f2e5a8679cd2dcfd6b30cdc48.svg
paper_authors.fullname: Tom Lippincott
paper_authors.type: user
paper_authors.user: tom-lippincott
paper_authors._id: 67f7bc4c03cfd3db7e6b75e4
paper_authors.name: Leshem Choshen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T13:27:27.281Z
paper_authors._id: 61bf40824b4300d0fb0acf59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
paper_authors.fullname: Leshem Choshen
paper_authors.type: user
paper_authors.user: borgr
paper_authors._id: 67f7bc4c03cfd3db7e6b75e5
paper_authors.name: Craig Messner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T13:27:19.225Z
paper_authors._id: 64b8556209e7f9e822603f1c
paper_authors.avatarUrl: /avatars/62925b4d9707fcff5b56f2903cced28b.svg
paper_authors.fullname: Craig Messner
paper_authors.type: user
paper_authors.user: cmessner
paper_title: Pretraining Language Models for Diachronic Linguistic Change Discovery
paper_summary: Large language models (LLMs) have shown potential as tools for scientific
  discovery. This has engendered growing interest in their use in humanistic
  disciplines, such as historical linguistics and literary studies. These fields
  often construct arguments on the basis of delineations like genre, or more
  inflexibly, time period. Although efforts have been made to restrict inference
  to specific domains via fine-tuning or model editing, we posit that the only
  true guarantee is domain-restricted pretraining -- typically, a data- and
  compute-expensive proposition.
    We show that efficient pretraining techniques can produce useful models over
  corpora too large for easy manual inspection but too small for "typical" LLM
  approaches. We employ a novel date-attribution pipeline in order to obtain a
  temporally-segmented dataset of five 10-million-word slices. We train two
  corresponding five-model batteries over these corpus segments, efficient
  pretraining and Llama3-8B parameter efficiently finetuned.
    We find that the pretrained models are faster to train than the finetuned
  baselines and that they better respect the historical divisions of our corpus.
  Emphasizing speed and precision over a-historical comprehensiveness enables a
  number of novel approaches to hypothesis discovery and testing in our target
  fields. Taking up diachronic linguistics as a testbed, we show that our method
  enables the detection of a diverse set of phenomena, including en masse lexical
  change, non-lexical (grammatical and morphological) change, and word sense
  introduction/obsolescence. We provide a ready-to-use pipeline that allows
  extension of our approach to other target fields with only minimal adaptation.
paper_ai_summary: Efficient pretraining techniques and temporally-segmented datasets produce faster and more historically accurate large language models, enhancing diachronic linguistic analysis.
paper_ai_keywords: efficient pretraining date-attribution pipeline temporally-segmented dataset parameter-efficient fine-tuning diachronic linguistics en masse lexical change non-lexical change word sense introduction/obsolescence