  Models trained to estimate word probabilities in context have become
ubiquitous in natural language processing. How do these models use lexical cues
in context to inform their word probabilities? To answer this question, we
present a case study analyzing the pre-trained BERT model with tests informed
by semantic priming. Using English lexical stimuli that show priming in humans,
we find that BERT too shows "priming," predicting a word with greater
probability when the context includes a related word versus an unrelated one.
This effect decreases as the amount of information provided by the context
increases. Follow-up analysis shows BERT to be increasingly distracted by
related prime words as context becomes more informative, assigning lower
probabilities to related words. Our findings highlight the importance of
considering contextual constraint effects when studying word prediction in
these models, and highlight possible parallels with human processing.

Computation and Language