The question of how to probe contextual word representations  in a way that is  principled and useful has seen significant recent attention . In our contribution to this discussion, we argue , first, for a probe metric that reflects the  trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments with such metrics show that probe's performance curves often fail to align with widely accepted rankings between language representations (with, e .g.,  non-contextual representations outperforming contextual ones). These results lead us to argue, second, that commonsimplistic probe taskssuch as POS labeling and dependency arc labeling, are inadequate to evaluate the properties encoded in contextual word representations. We propose full dependency parsing as an example probe task, and demonstrate it with the Pareto hypervolume . In support of our arguments, the results of this illustrative experiment conform closer to accepted rankings among contextual word representations.
The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature . In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments using Pareto hypervolume as an evaluation metric show that probes often do not conform to our expectations---e .g., why should the non-contextual fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech labeling and dependency arc labeling, are inadequate to evaluate the linguistic structure encoded in contextual word representations. This leads us to propose full dependency parsing as a probing task . In support of our suggestion that harder probing tasks are necessary, our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.
A
for linguistic structure
A
both
R
recent attention
attention recently in the NLP literature
D
, first,
A
fundamental
R
with such metrics show that probe's performance curves often fail to align with widely accepted rankings between language representations (with, e
using Pareto hypervolume as an evaluation metric show that probes often do not conform to our expectations---e
A
why should the
R
representations outperforming contextual ones). These results lead us to argue, second, that commonsimplistic probe taskssuch as POS
fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech
R
properties
linguistic structure
R
We
This leads us to
R
an example probe task, and demonstrate it with the Pareto hypervolume
a probing task
R
arguments, the results of this illustrative experiment conform closer to accepted rankings among contextual word
suggestion that harder probing tasks are necessary, our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual