Pretraining Language Models for Diachronic Linguistic Change Discovery
Large language models (LLMs) have shown potential as tools for scientific
discovery. This has engendered growing interest in their use in humanistic
disciplines, such as historical linguistics and literary studies. These fields
often construct arguments on the basis of delineations like genre, or more
inflexibly, time period. Although efforts have been made to restrict inference
to specific domains via fine-tuning or model editing, we posit that the only
true guarantee is domain-restricted pretraining -- typically, a data- and
compute-expensive proposition.
  We show that efficient pretraining techniques can produce useful models over
corpora too large for easy manual inspection but too small for "typical" LLM
approaches. We employ a novel date-attribution pipeline in order to obtain a
temporally-segmented dataset of five 10-million-word slices. We train two
corresponding five-model batteries over these corpus segments, efficient
pretraining and Llama3-8B parameter efficiently finetuned.
  We find that the pretrained models are faster to train than the finetuned
baselines and that they better respect the historical divisions of our corpus.
Emphasizing speed and precision over a-historical comprehensiveness enables a
number of novel approaches to hypothesis discovery and testing in our target
fields. Taking up diachronic linguistics as a testbed, we show that our method
enables the detection of a diverse set of phenomena, including en masse lexical
change, non-lexical (grammatical and morphological) change, and word sense
introduction/obsolescence. We provide a ready-to-use pipeline that allows
extension of our approach to other target fields with only minimal adaptation.
67f7bc4c03cfd3db7e6b75e1
Elisabeth Fittschen
claimed_verified
2025-04-10T15:45:39.921Z
6706d8a8d218bb754322d731
/avatars/807083029d5bae3cbafc05e03035b0a2.svg
Elisabeth Fittschen
user
efittschen
67f7bc4c03cfd3db7e6b75e2
Sabrina Li
67f7bc4c03cfd3db7e6b75e3
Tom Lippincott
admin_assigned
2025-04-10T13:27:35.414Z
63418125a7582111c3f57bca
/avatars/8128615f2e5a8679cd2dcfd6b30cdc48.svg
Tom Lippincott
user
tom-lippincott
67f7bc4c03cfd3db7e6b75e4
Leshem Choshen
admin_assigned
2025-04-10T13:27:27.281Z
61bf40824b4300d0fb0acf59
https://cdn-avatars.huggingface.co/v1/production/uploads/1644224872623-61bf40824b4300d0fb0acf59.jpeg
Leshem Choshen
user
borgr
67f7bc4c03cfd3db7e6b75e5
Craig Messner
admin_assigned
2025-04-10T13:27:19.225Z
64b8556209e7f9e822603f1c
/avatars/62925b4d9707fcff5b56f2903cced28b.svg
Craig Messner
user
cmessner
Pretraining Language Models for Diachronic Linguistic Change Discovery
Large language models (LLMs) have shown potential as tools for scientific
discovery. This has engendered growing interest in their use in humanistic
disciplines, such as historical linguistics and literary studies. These fields
often construct arguments on the basis of delineations like genre, or more
inflexibly, time period. Although efforts have been made to restrict inference
to specific domains via fine-tuning or model editing, we posit that the only
true guarantee is domain-restricted pretraining -- typically, a data- and
compute-expensive proposition.
  We show that efficient pretraining techniques can produce useful models over
corpora too large for easy manual inspection but too small for "typical" LLM
approaches. We employ a novel date-attribution pipeline in order to obtain a
temporally-segmented dataset of five 10-million-word slices. We train two
corresponding five-model batteries over these corpus segments, efficient
pretraining and Llama3-8B parameter efficiently finetuned.
  We find that the pretrained models are faster to train than the finetuned
baselines and that they better respect the historical divisions of our corpus.
Emphasizing speed and precision over a-historical comprehensiveness enables a
number of novel approaches to hypothesis discovery and testing in our target
fields. Taking up diachronic linguistics as a testbed, we show that our method
enables the detection of a diverse set of phenomena, including en masse lexical
change, non-lexical (grammatical and morphological) change, and word sense
introduction/obsolescence. We provide a ready-to-use pipeline that allows
extension of our approach to other target fields with only minimal adaptation.
Efficient pretraining techniques and temporally-segmented datasets produce faster and more historically accurate large language models, enhancing diachronic linguistic analysis.
efficient pretraining date-attribution pipeline temporally-segmented dataset parameter-efficient fine-tuning diachronic linguistics en masse lexical change non-lexical change word sense introduction/obsolescence