  Meanings of words change over time and across domains. Detecting the semantic
changes of words is an important task for various NLP applications that must
make time-sensitive predictions. We consider the problem of predicting whether
a given target word, $w$, changes its meaning between two different text
corpora, $\mathcal{C}_1$ and $\mathcal{C}_2$. For this purpose, we propose
$\textit{Swapping-based Semantic Change Detection}$ (SSCD), an unsupervised
method that randomly swaps contexts between $\mathcal{C}_1$ and $\mathcal{C}_2$
where $w$ occurs. We then look at the distribution of contextualised word
embeddings of $w$, obtained from a pretrained masked language model (MLM),
representing the meaning of $w$ in its occurrence contexts in $\mathcal{C}_1$
and $\mathcal{C}_2$. Intuitively, if the meaning of $w$ does not change between
$\mathcal{C}_1$ and $\mathcal{C}_2$, we would expect the distributions of
contextualised word embeddings of $w$ to remain the same before and after this
random swapping process. Despite its simplicity, we demonstrate that even by
using pretrained MLMs without any fine-tuning, our proposed context swapping
method accurately predicts the semantic changes of words in four languages
(English, German, Swedish, and Latin) and across different time spans (over 50
years and about five years). Moreover, our method achieves significant
performance improvements compared to strong baselines for the English semantic
change prediction task. Source code is available at
https://github.com/a1da4/svp-swap .

Computation and Language