1 Since the task's introduction, there has been no formal consensus around which of the two approaches offers the better cost-benefit tradeoff: precise (at risk of being impractical), or organic (at risk of being ill-defined).

the prevailing assumption across annotation methods is that there is a single ''true'' inference about h given p that we should train models to predict, and that this label can be approximated by aggregating multiple (possibly noisy) human ratings as is typical in many other labelling tasks (Snow et al., 2008;Callison-Burch and Dredze, 2010).


Our goal is to build S xy such that its sentences containing x are ''highly characteristic'' of x's shared meaning with y, and vice versa.

our primary question is whether automatically extracted PSTS sentences for a paraphrase pair truly reflect the paraphrase meaning.


Our method for extracting sentences for PSTS is inspired by bilingual pivoting (Bannard and Callison-Burch, 2005), which discovers samelanguage paraphrases by ''pivoting'' over bilingual parallel corpora.

we use BERT in its configuration for sentence pair classification tasks, where the input consists of two tokenized sentences (c t and c w ), preceded by a [CLS] token and separated by a [SEP] token.