  Since Pretrained Language Models (PLMs) are the cornerstone of the most
recent Information Retrieval (IR) models, the way they encode semantic
knowledge is particularly important. However, little attention has been given
to studying the PLMs' capability to capture hierarchical semantic knowledge.
Traditionally, evaluating such knowledge encoded in PLMs relies on their
performance on a task-dependent evaluation approach based on proxy tasks, such
as hypernymy detection. Unfortunately, this approach potentially ignores other
implicit and complex taxonomic relations. In this work, we propose a
task-agnostic evaluation method able to evaluate to what extent PLMs can
capture complex taxonomy relations, such as ancestors and siblings. The
evaluation is based on intrinsic properties that capture the hierarchical
nature of taxonomies. Our experimental evaluation shows that the
lexico-semantic knowledge implicitly encoded in PLMs does not always capture
hierarchical relations. We further demonstrate that the proposed properties can
be injected into PLMs to improve their understanding of hierarchy. Through
evaluations on taxonomy reconstruction, hypernym discovery and reading
comprehension tasks, we show that the knowledge about hierarchy is moderately
but not systematically transferable across tasks.

Computation and Language;Information Retrieval