  In recent years, distributional language representation models have
demonstrated great practical success. At the same time, the need for
interpretability has elicited questions on their intrinsic properties and
capabilities. Crucially, distributional models are often inconsistent when
dealing with compositional phenomena in natural language, which has significant
implications for their safety and fairness. Despite this, most current research
on compositionality is directed towards improving their performance on
similarity tasks only. This work takes a different approach, and proposes a
methodology for measuring compositional behavior in contemporary language
models. Specifically, we focus on adjectival modifier phenomena in
adjective-noun phrases. We introduce three novel tests of compositional
behavior inspired by Montague semantics. Our experimental results indicate that
current neural language models behave according to the expected linguistic
theories to a limited extent only. This raises the question of whether these
language models are not able to capture the semantic properties we evaluated,
or whether linguistic theories from Montagovian tradition would not match the
expected capabilities of distributional models.

Computation and Language