title: Time is Encoded in the Weights of Finetuned Language Models
summary: We present time vectors, a simple tool to customize language models to new
  time periods. Time vectors are created by finetuning a language model on data
  from a single time (e.g., a year or month), and then subtracting the weights of
  the original pretrained model. This vector specifies a direction in weight
  space that, as our experiments show, improves performance on text from that
  time period. Time vectors specialized to adjacent time periods appear to be
  positioned closer together in a manifold. Using this structure, we interpolate
  between time vectors to induce new models that perform better on intervening
  and future time periods, without any additional training. We demonstrate the
  consistency of our findings across different tasks, domains, model sizes, and
  time scales. Our results suggest that time is encoded in the weight space of
  finetuned models.
paper_authors._id: 65850bd6b476868782d5379a
paper_authors.name: Kai Nylund
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T11:03:03.126Z
paper_authors._id: 6260b95e4fe7f6521dd6a083
paper_authors.avatarUrl: /avatars/f3e20c204b83199507646604a30660bb.svg
paper_authors.fullname: Kai Nylund
paper_authors.type: user
paper_authors.user: KaiNylund
paper_authors._id: 65850bd6b476868782d5379b
paper_authors.name: Suchin Gururangan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T11:03:10.681Z
paper_authors._id: 5e9272254957053f60648a01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
paper_authors.fullname: Suchin Gururangan
paper_authors.type: user
paper_authors.user: suching
paper_authors._id: 65850bd6b476868782d5379c
paper_authors.name: Noah A. Smith
paper_title: Time is Encoded in the Weights of Finetuned Language Models
paper_summary: We present time vectors, a simple tool to customize language models to new
  time periods. Time vectors are created by finetuning a language model on data
  from a single time (e.g., a year or month), and then subtracting the weights of
  the original pretrained model. This vector specifies a direction in weight
  space that, as our experiments show, improves performance on text from that
  time period. Time vectors specialized to adjacent time periods appear to be
  positioned closer together in a manifold. Using this structure, we interpolate
  between time vectors to induce new models that perform better on intervening
  and future time periods, without any additional training. We demonstrate the
  consistency of our findings across different tasks, domains, model sizes, and
  time scales. Our results suggest that time is encoded in the weight space of
  finetuned models.
paper_ai_summary: Time vectors, derived by fine-tuning language models on specific time periods, enhance performance on text from those periods and can be interpolated to improve performance on future periods without further training.
paper_ai_keywords: time vectors finetuning weight space interpolation performance language models time periods manifold weight space time encoding