Zipfian Whitening
The word embedding space in neural models is skewed, and correcting this can
improve task performance. We point out that most approaches for modeling,
correcting, and measuring the symmetry of an embedding space implicitly assume
that the word frequencies are uniform; in reality, word frequencies follow a
highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
performing PCA whitening weighted by the empirical word frequency that follows
Zipf's law significantly improves task performance, surpassing established
baselines. From a theoretical perspective, both our approach and existing
methods can be clearly categorized: word representations are distributed
according to an exponential family with either uniform or Zipfian base
measures. By adopting the latter approach, we can naturally emphasize
informative low-frequency words in terms of their vector norm, which becomes
evident from the information-geometric perspective, and in terms of the loss
functions for imbalanced classification. Additionally, our theory corroborates
that popular natural language processing methods, such as skip-gram negative
sampling, WhiteningBERT, and headless language models, work well just because
their word embeddings encode the empirical word frequency into the underlying
probabilistic model.
6728baa54fb44fa37e300a46
Sho Yokoi
admin_assigned
2024-11-04T21:36:21.297Z
66c5d59e55c196259b54e383
/avatars/708f54d2888885f07d6db1505712a173.svg
Sho Yokoi
user
eumesy
6728baa54fb44fa37e300a47
Han Bao
6728baa54fb44fa37e300a48
Hiroto Kurita
admin_assigned
2024-11-04T21:36:36.823Z
62e77c8e52d9e64282174a3c
/avatars/5978db338fa82b08d3a485d72d7813b2.svg
Hiroto Kurita
user
kurita
6728baa54fb44fa37e300a49
Hidetoshi Shimodaira
admin_assigned
2024-11-04T21:36:30.757Z
6416db0df1e8690893589e37
/avatars/411dd24ebaca270cdd5e6fe0962aacd2.svg
Hidetoshi Shimodaira
user
shimosan
Zipfian Whitening
The word embedding space in neural models is skewed, and correcting this can
improve task performance. We point out that most approaches for modeling,
correcting, and measuring the symmetry of an embedding space implicitly assume
that the word frequencies are uniform; in reality, word frequencies follow a
highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
performing PCA whitening weighted by the empirical word frequency that follows
Zipf's law significantly improves task performance, surpassing established
baselines. From a theoretical perspective, both our approach and existing
methods can be clearly categorized: word representations are distributed
according to an exponential family with either uniform or Zipfian base
measures. By adopting the latter approach, we can naturally emphasize
informative low-frequency words in terms of their vector norm, which becomes
evident from the information-geometric perspective, and in terms of the loss
functions for imbalanced classification. Additionally, our theory corroborates
that popular natural language processing methods, such as skip-gram negative
sampling, WhiteningBERT, and headless language models, work well just because
their word embeddings encode the empirical word frequency into the underlying
probabilistic model.
PCA whitening based on empirical word frequency improves task performance in neural models, particularly by emphasizing informative low-frequency words.
word embedding space PCA whitening Zipf's law exponential family information-geometric perspective skip-gram negative sampling WhiteningBERT headless language models