title: Zipfian Whitening
summary: The word embedding space in neural models is skewed, and correcting this can
  improve task performance. We point out that most approaches for modeling,
  correcting, and measuring the symmetry of an embedding space implicitly assume
  that the word frequencies are uniform; in reality, word frequencies follow a
  highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
  performing PCA whitening weighted by the empirical word frequency that follows
  Zipf's law significantly improves task performance, surpassing established
  baselines. From a theoretical perspective, both our approach and existing
  methods can be clearly categorized: word representations are distributed
  according to an exponential family with either uniform or Zipfian base
  measures. By adopting the latter approach, we can naturally emphasize
  informative low-frequency words in terms of their vector norm, which becomes
  evident from the information-geometric perspective, and in terms of the loss
  functions for imbalanced classification. Additionally, our theory corroborates
  that popular natural language processing methods, such as skip-gram negative
  sampling, WhiteningBERT, and headless language models, work well just because
  their word embeddings encode the empirical word frequency into the underlying
  probabilistic model.
paper_authors._id: 6728baa54fb44fa37e300a46
paper_authors.name: Sho Yokoi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:36:21.297Z
paper_authors._id: 66c5d59e55c196259b54e383
paper_authors.avatarUrl: /avatars/708f54d2888885f07d6db1505712a173.svg
paper_authors.fullname: Sho Yokoi
paper_authors.type: user
paper_authors.user: eumesy
paper_authors._id: 6728baa54fb44fa37e300a47
paper_authors.name: Han Bao
paper_authors._id: 6728baa54fb44fa37e300a48
paper_authors.name: Hiroto Kurita
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:36:36.823Z
paper_authors._id: 62e77c8e52d9e64282174a3c
paper_authors.avatarUrl: /avatars/5978db338fa82b08d3a485d72d7813b2.svg
paper_authors.fullname: Hiroto Kurita
paper_authors.type: user
paper_authors.user: kurita
paper_authors._id: 6728baa54fb44fa37e300a49
paper_authors.name: Hidetoshi Shimodaira
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:36:30.757Z
paper_authors._id: 6416db0df1e8690893589e37
paper_authors.avatarUrl: /avatars/411dd24ebaca270cdd5e6fe0962aacd2.svg
paper_authors.fullname: Hidetoshi Shimodaira
paper_authors.type: user
paper_authors.user: shimosan
paper_title: Zipfian Whitening
paper_summary: The word embedding space in neural models is skewed, and correcting this can
  improve task performance. We point out that most approaches for modeling,
  correcting, and measuring the symmetry of an embedding space implicitly assume
  that the word frequencies are uniform; in reality, word frequencies follow a
  highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
  performing PCA whitening weighted by the empirical word frequency that follows
  Zipf's law significantly improves task performance, surpassing established
  baselines. From a theoretical perspective, both our approach and existing
  methods can be clearly categorized: word representations are distributed
  according to an exponential family with either uniform or Zipfian base
  measures. By adopting the latter approach, we can naturally emphasize
  informative low-frequency words in terms of their vector norm, which becomes
  evident from the information-geometric perspective, and in terms of the loss
  functions for imbalanced classification. Additionally, our theory corroborates
  that popular natural language processing methods, such as skip-gram negative
  sampling, WhiteningBERT, and headless language models, work well just because
  their word embeddings encode the empirical word frequency into the underlying
  probabilistic model.
paper_ai_keywords: word embedding space neural models task performance symmetry PCA whitening Zipf's law information-geometric perspective exponential family uniform base measures Zipfian base measures vector norm loss functions imbalanced classification skip-gram negative sampling WhiteningBERT headless language models probabilistic model