text:
\subsection{Discussions}
We showed that our metric combining contextualized
embeddings and
Earth Mover's Distance outperforms strong unsupervised metrics on 3 out of 4 tasks, i.e., METEOR++ on machine translation by 5.7 points, SPICE on image captioning by 3.0 points, and METEOR on dialogue response generation by 2.2 points. The best correlation we achieved is combining contextualized word embeddings and WMD, which even rivals or exceeds SOTA task-dependent \emph{supervised} metrics across  different tasks. Especially in machine translation, our word mover metric pushes correlations in machine translation to 74.3 on average (5.8 points over the SOTA supervised metric and 2.4 points over contemporaneous BERTScore). The major improvements come from contextualized BERT embeddings rather than word2vec and ELMo, and from fine-tuning BERT on large NLI datasets. However, we also observed that soft alignments (MoverScore) marginally outperforms hard alignments (BERTScore).
Regarding the effect of $n$-grams in word mover metrics,
unigrams slightly outperforms
bigrams on average.
For the effect of aggregation functions, we suggested
effective techniques for layer-wise consolidations, namely $p$-means and routing, both of which are close to the performance of the best layer and on par with each other (see the appendix).

\section{Introduction}\label{sec:introduction}
\input{introduction.tex}

\section{Related Work}\label{sec:related}
\input{relatedwork.tex}

\input{model.tex}

\section{Empirical Evaluation}
\input{results.tex}\label{sec:results}

\section{Conclusion}\label{sec:conclusion}
\input{conclusion.tex}

\section*{Acknowledgments}
We thank the anonymous reviewers for their comments, which greatly improved the final version of the paper.
This work has been supported by the German Research Foundation as part of the Research Training
Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) at the Technische
Universit\"at Darmstadt under grant No. GRK 1994/1.
Fei Liu is supported in part by NSF grant IIS-1909603.

\section{Our MoverScore Meric}
We have motivated the need for better metrics capable of evaluating disparate NLG tasks.
We now describe our metric, namely MoverScore, built upon a combination of (i) contextualized representations of system and reference texts and (ii) a distance between these representations measuring the semantic distance between system outputs and references.
It is particularly important for a metric to not only capture the amount of \emph{shared content} between two texts, i.e., \textsf{\small intersect(A,B)}, as is the case with many semantic textual similarity measures~\cite{Peters:2018,Devlin:2018};
but also to accurately reflect to what extent the system text has \emph{deviated} from the reference, i.e., \textsf{\small union(A,B)} - \textsf{\small intersect(A,B)}, which is the intuition behind using a distance metric.
\subsection{Measuring Semantic Distance}
Let $\boldsymbol{x} = ({\textnormal{x}}_1, \dots, {\textnormal{x}}_m)$ be a sentence viewed as a sequence of words.
We denote by $\boldsymbol{x}^n$ the sequence
of $n$-grams of $\boldsymbol{x}$ (i.e., $\boldsymbol{x}^1=\boldsymbol{x}$ is the sequence
of words and $\boldsymbol{x}^2$ is the sequence of bigrams).
Furthermore, let $\boldsymbol{f}_{\boldsymbol{x}^n}\in\mathbb{R}_+^{|\boldsymbol{x}^n|}$ be a
vector
of weights,
one weight for each
$n$-gram of $\boldsymbol{x}^n$.
We can assume $\boldsymbol{f}_{\boldsymbol{x}^n}^{\mathsf{T}}\boldsymbol{1}=1$, making $\boldsymbol{f}_{\boldsymbol{x}^n}$ a distribution over $n$-grams. Intuitively, the effect of some $n$-grams like those including function words can be downplayed by giving them lower weights, e.g., using Inverse Document Frequency (IDF).

Word Mover's Distance (WMD)~\cite{Kusner:2015}, a special case of Earth Mover's Distance~\cite{Rubner2000}, measures semantic distance between texts by aligning semantically similar words and finding
the amount of flow traveling between these words.
It was shown
useful for text classification and textual similarity tasks~\cite{Kusner:2015}.
Here, we formulate a generalization operating on $n$-grams.
Let $\boldsymbol{x}$ and $\boldsymbol{y}$ be two sentences viewed as sequences of $n$-grams: $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$. If we have a distance metric $d$ between $n$-grams, then we can define the \emph{transportation cost} matrix ${\bm{C}}$ such that ${\bm{C}}_{ij} = d({\textnormal{x}}^n_i, {\textnormal{y}}^n_j)$ is the distance between the $i$-th $n$-gram of $\boldsymbol{x}$ and the $j$-th $n$-gram of $\boldsymbol{y}$.
The $\text{WMD}$ between the two sequences
of $n$-grams $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$ with associated $n$-gram weights $\boldsymbol{f}_{\boldsymbol{x}^n}$ and $\boldsymbol{f}_{\boldsymbol{y}^n}$ is then given by:
\begin{align*}
& \text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := \min_{{\bm{F}} \in\mathbb{R}^{|\boldsymbol{x}^n|\times |\boldsymbol{y}^n|}} \langle {\bm{C}}, {\bm{F}} \rangle, \\
&\text{s.t. } {\bm{F}}\boldsymbol{1}=\boldsymbol{f}_{\boldsymbol{x}^n},\;\;{\bm{F}}^{\intercal}\boldsymbol{1}=\boldsymbol{f}_{\boldsymbol{y}^n}.
\end{align*}
where ${\bm{F}}$ is the \emph{transportation flow} matrix with ${\bm{F}}_{ij}$ denoting the amount of flow traveling from the $i$-th $n$-gram ${\textnormal{x}}^n_i$ in $\boldsymbol{x}^n$ to the $j$-th $n$-gram ${\textnormal{y}}^n_j$ in $\boldsymbol{y}^n$.
Here, $\langle {\bm{C}}, {\bm{F}} \rangle$ denotes the sum of all matrix entries of the matrix ${\bm{C}}\odot {\bm{F}}$, where $\odot$ denotes element-wise multiplication.
Then $\text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n)$ is the minimal transportation cost between $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$ where $n$-grams are weighted by $\boldsymbol{f}_{\boldsymbol{x}^n}$ and $\boldsymbol{f}_{\boldsymbol{y}^n}$.

In practice, we compute the Euclidean distance between the embedding representations of $n$-grams:
$d({\textnormal{x}}^n_i, {\textnormal{y}}^n_j) = ||E({\textnormal{x}}^n_i) - E({\textnormal{y}}^n_j)||_2$ where
$E$ is the embedding function which maps an $n$-gram to its vector representation. Usually, \emph{static} word embeddings like word2vec are used to compute $E$ but these cannot capture word order or compositionality. Alternatively, we investigate contextualized embeddings like ELMo and BERT because they encode information about the whole sentence into each word vector.

We compute the $n$-gram embeddings as the weighted sum over its word embeddings.
Formally, if $\boldsymbol{x}^n_i = ({\textnormal{x}}_i, \dots, {\textnormal{x}}_{i+n-1})$ is the $i$-th $n$-gram from sentence $\boldsymbol{x}$, its embedding is given by:
\begin{equation}
\label{eq:n-gram-embedding}
E(\boldsymbol{x}^n_i) = \sum_{k=i}^{i+n-1} {\rm idf}({\textnormal{x}}_k)\cdot E({\textnormal{x}}_k)
\end{equation}
where ${\rm idf}({\textnormal{x}}_k)$ is the IDF of word ${\textnormal{x}}_k$ computed from all sentences in the corpus and $E({\textnormal{x}}_k)$ is its word vector.
Furthermore, the weight associated to the $n$-gram
$\boldsymbol{x}^n_i$ is given by:
\begin{equation}
\boldsymbol{f}_{\boldsymbol{x}^n_i} = \frac{1}{Z} \sum_{k=i}^{i+n-1} {\rm idf}({\textnormal{x}}_k)
\end{equation}
where $Z$ is a normalizing constant s.t. $\boldsymbol{f}_{\boldsymbol{x}^n}^{\mathsf{T}}\boldsymbol{1}=1$,

In the limiting case where $n$ is larger than the sentence's size, $\boldsymbol{x}^n$ contains only one $n$-gram: the whole sentence. Then $\text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n)$ reduces to computing the distance between the two sentence embeddings, namely Sentence Mover's Distance (SMD), denoted as:
\begin{align*}
\text{SMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := ||E(\boldsymbol{x}_1^{\l_x})-E(\boldsymbol{y}_1^{\l_y})||
\end{align*}
where $\l_{x}$ and $\l_{y}$ are the size of sentences.

\paragraph{Hard and Soft Alignments}\quad
In contemporaneous work, BERTScore \citep{zhang:2019} also
models
the semantic distance between system and reference texts
for
evaluating text generation systems.
As shown in Figure \ref{fig:bertscore-wmd},
BERTScore (precision/recall) can be intuitively viewed as hard alignments (one-to-one) for words in a sentence pair, where each word in one sequence
travels to the most semantically similar word in the other sequence. In contrast,
MoverScore
goes beyond BERTScore as it relies on soft alignments (many-to-one) and allows to map semantically related words in one sequence to the respective word in the other sequence by solving a constrained optimization problem: finding the minimum
effort
to transform between two texts.

The formulation of Word Mover's Distance provides an important possibility to bias the metric towards precision or recall by using an asymmetric transportation cost matrix, which bridges a gap between MoverScore and BERTScore:
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/wmd-bertscore.pdf}
\caption{An illustration of MoverScore and BERTScore.
}
 \label{fig:bertscore-wmd}
\vspace{-0.2in}
\end{figure}

\begin{prop}\label{prop:main}
BERTScore (precision/recall) can be represented as a (non-optimized) Mover Distance
$\langle {\bm{C}}, {\bm{F}} \rangle$,
where ${\bm{C}}$ is a transportation cost matrix based on BERT and ${\bm{F}}$ is a
uniform
transportation flow matrix.\footnote{See the proof in the appendix.}
\end{prop}

\subsection{Contextualized Representations}

The task formulation naturally lends itself to deep contextualized representations for inducing word vectors $E({\textnormal{x}}_i)$.
Despite the recent success of multi-layer attentive neural architectures~\cite{Devlin:2018,Peters:2018}, consolidating layer-wise information remains an open problem
as different layers capture information at disparate scales and task-specific layer selection methods may be limited~\cite{Liu:2018:EMNLP,liu:2019}. \citet{tenney:2019} found that
a scalar mix of output layers trained from task-dependent supervisions would be effective in a deep transformer-based model. Instead, we investigate aggregation functions to consolidate layer-wise information, forming stationary representations of words without
supervision.

Consider a sentence $\boldsymbol{x}$ passed through contextualized encoders such as ELMo and BERT with $L$ layers. Each layer of the encoders produces a vector representation for each word ${\textnormal{x}}_i$ in $\boldsymbol{x}$.
We denote by  $\boldsymbol{z}_{i,l} \in \mathbb{R}^d$ the representation given by layer $l$, a $d$-dimensional vector.
Overall,
${\textnormal{x}}_i$ receives $L$ different vectors $(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})$.
An aggregation $\phi$ maps these $L$ vectors to one final vector:
\begin{equation*}
E({\textnormal{x}}_i) = \phi(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})
\numberthis
\end{equation*}
where $E({\textnormal{x}}_i)$ is the aggregated representation of the word ${\textnormal{x}}_i$.

We study two alternatives for $\phi$:
(i) the concatenation of power means \cite{Rckl2018ConcatenatedPW} as a generalized pooling mechanism, and
(ii) a routing mechanism for aggregation~\cite{zhao:2018,zhao-etal-2019-towards}.
We relegate the routing method to appendix, as it does not yield better results than power means.

\paragraph{Power Means}
Power means
is an effective
generalization of pooling techniques for aggregating information.
It computes a non-linear average of a set of values with an exponent $p$ (Eq.~(\ref{eq:power_means})).
Following \citet{Rckl2018ConcatenatedPW}, we exploit power means to aggregate vector representations $(\boldsymbol{z}_{i,l})_{l=1}^L$ pertaining to the $i$-th word from all layers of a deep neural architecture.
Let $p\in\mathbb{R}\cup\{\pm\infty\}$, the $p$-mean of $(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})$ is:
\begin{align*}

  \boldsymbol{h}_i^{(p)}=\left(\frac{\boldsymbol{z}_{i,1}^p+\cdots+\boldsymbol{z}_{i,L}^p}{L}\right)^{1/p} \in \mathbb{R}^
  \label{eq:power_means}
  \numberthis
\end{align*}
where
exponentiation is applied elementwise.
This generalized form can induce common named means such as arithmetic mean ($p=1$) and geometric mean ($p=0$). In extreme cases, a power mean reduces to the minimum value of the set when $p=-\infty$, and the maximum value when $p=+\infty$. The concatenation of $p$-mean vectors we use in this paper is denoted by:
\begin{align*}
  E(x_i) =
  \mathbf{h}_i^{(p_1)}\oplus\cdots\oplus \mathbf{h}_i^{(p_K)}
  \numberthis
\end{align*}
where $\oplus$
is vector concatenation;
$\{p_1,\ldots,p_K\}$ are exponent values, and we use $K=3$ with
$p=1,\pm\infty$
in this work.

\subsection{Summary of MoverScore Variations}
We investigate our MoverScore along four dimensions:
(i) the granularity of embeddings, i.e., the size of $n$ for $n$-grams, (ii) the choice of pretrained embedding mechanism, (iii) the fine-tuning task used for BERT\footnote{ELMo usually requires heavy layers on the top, which restricts the power of fine-tuning tasks for ELMo.} (iv) the aggregation technique ($p$-means or routing) when applicable.

\paragraph{Granularity}
We used $n = 1$ and $n = 2$ as well as full sentences ($n=$ size of the sentence).

\paragraph{Embedding Mechanism}
We obtained word embeddings from three different methods: \emph{static} embedding with word2vec as well as contextualized embedding with ELMo and BERT. If $n > 1$, $n$-gram embeddings are calculated by Eq.~(\ref{eq:n-gram-embedding}). Note that they represent sentence embeddings when $n=$ size of the sentence.

\paragraph{Fine-tuning Tasks}
Natural Language Inference (NLI) and paraphrasing pose high demands in understanding sentence meaning.
This motivated us to
fine-tune BERT representations on two NLI datasets, MultiNLI and QANLI, and one Paraphrase dataset, QQP---the largest datasets in GLUE \cite{wang:2018}.
We fine-tune BERT on each of these, yielding different contextualized embeddings for our general evaluation metrics.

\paragraph{Aggregation}
For ELMo, we aggregate
word representations given by all
three ELMo layers, using
$p$-means or routing (see the appendix).
Word representations in BERT are aggregated from the last five layers, using $p$-means or routing since the representations in the initial layers are
less suited
for use in
downstream tasks \cite{liu:2019}.

\subsection{Discussions}
We showed that our metric combining contextualized
embeddings and
Earth Mover's Distance outperforms strong unsupervised metrics on 3 out of 4 tasks, i.e., METEOR++ on machine translation by 5.7 points, SPICE on image captioning by 3.0 points, and METEOR on dialogue response generation by 2.2 points. The best correlation we achieved is combining contextualized word embeddings and WMD, which even rivals or exceeds SOTA task-dependent \emph{supervised} metrics across  different tasks. Especially in machine translation, our word mover metric pushes correlations in machine translation to 74.3 on average (5.8 points over the SOTA supervised metric and 2.4 points over contemporaneous BERTScore). The major improvements come from contextualized BERT embeddings rather than word2vec and ELMo, and from fine-tuning BERT on large NLI datasets. However, we also observed that soft alignments (MoverScore) marginally outperforms hard alignments (BERTScore).
Regarding the effect of $n$-grams in word mover metrics,
unigrams slightly outperforms
bigrams on average.
For the effect of aggregation functions, we suggested
effective techniques for layer-wise consolidations, namely $p$-means and routing, both of which are close to the performance of the best layer and on par with each other (see the appendix).

\section{Introduction}\label{sec:introduction}
\input{introduction.tex}

\section{Related Work}\label{sec:related}
\input{relatedwork.tex}

\input{model.tex}

\section{Empirical Evaluation}
\input{results.tex}\label{sec:results}

\section{Conclusion}\label{sec:conclusion}
\input{conclusion.tex}

\section*{Acknowledgments}
We thank the anonymous reviewers for their comments, which greatly improved the final version of the paper.
This work has been supported by the German Research Foundation as part of the Research Training
Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) at the Technische
Universit\"at Darmstadt under grant No. GRK 1994/1.
Fei Liu is supported in part by NSF grant IIS-1909603.

\section{Our MoverScore Meric}
We have motivated the need for better metrics capable of evaluating disparate NLG tasks.
We now describe our metric, namely MoverScore, built upon a combination of (i) contextualized representations of system and reference texts and (ii) a distance between these representations measuring the semantic distance between system outputs and references.
It is particularly important for a metric to not only capture the amount of \emph{shared content} between two texts, i.e., \textsf{\small intersect(A,B)}, as is the case with many semantic textual similarity measures~\cite{Peters:2018,Devlin:2018};
but also to accurately reflect to what extent the system text has \emph{deviated} from the reference, i.e., \textsf{\small union(A,B)} - \textsf{\small intersect(A,B)}, which is the intuition behind using a distance metric.
\subsection{Measuring Semantic Distance}
Let $\boldsymbol{x} = ({\textnormal{x}}_1, \dots, {\textnormal{x}}_m)$ be a sentence viewed as a sequence of words.
We denote by $\boldsymbol{x}^n$ the sequence
of $n$-grams of $\boldsymbol{x}$ (i.e., $\boldsymbol{x}^1=\boldsymbol{x}$ is the sequence
of words and $\boldsymbol{x}^2$ is the sequence of bigrams).
Furthermore, let $\boldsymbol{f}_{\boldsymbol{x}^n}\in\mathbb{R}_+^{|\boldsymbol{x}^n|}$ be a
vector
of weights,
one weight for each
$n$-gram of $\boldsymbol{x}^n$.
We can assume $\boldsymbol{f}_{\boldsymbol{x}^n}^{\mathsf{T}}\boldsymbol{1}=1$, making $\boldsymbol{f}_{\boldsymbol{x}^n}$ a distribution over $n$-grams. Intuitively, the effect of some $n$-grams like those including function words can be downplayed by giving them lower weights, e.g., using Inverse Document Frequency (IDF).

Word Mover's Distance (WMD)~\cite{Kusner:2015}, a special case of Earth Mover's Distance~\cite{Rubner2000}, measures semantic distance between texts by aligning semantically similar words and finding
the amount of flow traveling between these words.
It was shown
useful for text classification and textual similarity tasks~\cite{Kusner:2015}.
Here, we formulate a generalization operating on $n$-grams.
Let $\boldsymbol{x}$ and $\boldsymbol{y}$ be two sentences viewed as sequences of $n$-grams: $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$. If we have a distance metric $d$ between $n$-grams, then we can define the \emph{transportation cost} matrix ${\bm{C}}$ such that ${\bm{C}}_{ij} = d({\textnormal{x}}^n_i, {\textnormal{y}}^n_j)$ is the distance between the $i$-th $n$-gram of $\boldsymbol{x}$ and the $j$-th $n$-gram of $\boldsymbol{y}$.
The $\text{WMD}$ between the two sequences
of $n$-grams $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$ with associated $n$-gram weights $\boldsymbol{f}_{\boldsymbol{x}^n}$ and $\boldsymbol{f}_{\boldsymbol{y}^n}$ is then given by:
\begin{align*}
& \text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := \min_{{\bm{F}} \in\mathbb{R}^{|\boldsymbol{x}^n|\times |\boldsymbol{y}^n|}} \langle {\bm{C}}, {\bm{F}} \rangle, \\
&\text{s.t. } {\bm{F}}\boldsymbol{1}=\boldsymbol{f}_{\boldsymbol{x}^n},\;\;{\bm{F}}^{\intercal}\boldsymbol{1}=\boldsymbol{f}_{\boldsymbol{y}^n}.
\end{align*}
where ${\bm{F}}$ is the \emph{transportation flow} matrix with ${\bm{F}}_{ij}$ denoting the amount of flow traveling from the $i$-th $n$-gram ${\textnormal{x}}^n_i$ in $\boldsymbol{x}^n$ to the $j$-th $n$-gram ${\textnormal{y}}^n_j$ in $\boldsymbol{y}^n$.
Here, $\langle {\bm{C}}, {\bm{F}} \rangle$ denotes the sum of all matrix entries of the matrix ${\bm{C}}\odot {\bm{F}}$, where $\odot$ denotes element-wise multiplication.
Then $\text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n)$ is the minimal transportation cost between $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$ where $n$-grams are weighted by $\boldsymbol{f}_{\boldsymbol{x}^n}$ and $\boldsymbol{f}_{\boldsymbol{y}^n}$.

In practice, we compute the Euclidean distance between the embedding representations of $n$-grams:
$d({\textnormal{x}}^n_i, {\textnormal{y}}^n_j) = ||E({\textnormal{x}}^n_i) - E({\textnormal{y}}^n_j)||_2$ where
$E$ is the embedding function which maps an $n$-gram to its vector representation. Usually, \emph{static} word embeddings like word2vec are used to compute $E$ but these cannot capture word order or compositionality. Alternatively, we investigate contextualized embeddings like ELMo and BERT because they encode information about the whole sentence into each word vector.

We compute the $n$-gram embeddings as the weighted sum over its word embeddings.
Formally, if $\boldsymbol{x}^n_i = ({\textnormal{x}}_i, \dots, {\textnormal{x}}_{i+n-1})$ is the $i$-th $n$-gram from sentence $\boldsymbol{x}$, its embedding is given by:
\begin{equation}
\label{eq:n-gram-embedding}
E(\boldsymbol{x}^n_i) = \sum_{k=i}^{i+n-1} {\rm idf}({\textnormal{x}}_k)\cdot E({\textnormal{x}}_k)
\end{equation}
where ${\rm idf}({\textnormal{x}}_k)$ is the IDF of word ${\textnormal{x}}_k$ computed from all sentences in the corpus and $E({\textnormal{x}}_k)$ is its word vector.
Furthermore, the weight associated to the $n$-gram
$\boldsymbol{x}^n_i$ is given by:
\begin{equation}
\boldsymbol{f}_{\boldsymbol{x}^n_i} = \frac{1}{Z} \sum_{k=i}^{i+n-1} {\rm idf}({\textnormal{x}}_k)
\end{equation}
where $Z$ is a normalizing constant s.t. $\boldsymbol{f}_{\boldsymbol{x}^n}^{\mathsf{T}}\boldsymbol{1}=1$,

In the limiting case where $n$ is larger than the sentence's size, $\boldsymbol{x}^n$ contains only one $n$-gram: the whole sentence. Then $\text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n)$ reduces to computing the distance between the two sentence embeddings, namely Sentence Mover's Distance (SMD), denoted as:
\begin{align*}
\text{SMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := ||E(\boldsymbol{x}_1^{\l_x})-E(\boldsymbol{y}_1^{\l_y})||
\end{align*}
where $\l_{x}$ and $\l_{y}$ are the size of sentences.

\paragraph{Hard and Soft Alignments}\quad
In contemporaneous work, BERTScore \citep{zhang:2019} also
models
the semantic distance between system and reference texts
for
evaluating text generation systems.
As shown in Figure \ref{fig:bertscore-wmd},
BERTScore (precision/recall) can be intuitively viewed as hard alignments (one-to-one) for words in a sentence pair, where each word in one sequence
travels to the most semantically similar word in the other sequence. In contrast,
MoverScore
goes beyond BERTScore as it relies on soft alignments (many-to-one) and allows to map semantically related words in one sequence to the respective word in the other sequence by solving a constrained optimization problem: finding the minimum
effort
to transform between two texts.

The formulation of Word Mover's Distance provides an important possibility to bias the metric towards precision or recall by using an asymmetric transportation cost matrix, which bridges a gap between MoverScore and BERTScore:
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figures/wmd-bertscore.pdf}
\caption{An illustration of MoverScore and BERTScore.
}
 \label{fig:bertscore-wmd}
\vspace{-0.2in}
\end{figure}

\begin{prop}\label{prop:main}
BERTScore (precision/recall) can be represented as a (non-optimized) Mover Distance
$\langle {\bm{C}}, {\bm{F}} \rangle$,
where ${\bm{C}}$ is a transportation cost matrix based on BERT and ${\bm{F}}$ is a
uniform
transportation flow matrix.\footnote{See the proof in the appendix.}
\end{prop}

\subsection{Contextualized Representations}

The task formulation naturally lends itself to deep contextualized representations for inducing word vectors $E({\textnormal{x}}_i)$.
Despite the recent success of multi-layer attentive neural architectures~\cite{Devlin:2018,Peters:2018}, consolidating layer-wise information remains an open problem
as different layers capture information at disparate scales and task-specific layer selection methods may be limited~\cite{Liu:2018:EMNLP,liu:2019}. \citet{tenney:2019} found that
a scalar mix of output layers trained from task-dependent supervisions would be effective in a deep transformer-based model. Instead, we investigate aggregation functions to consolidate layer-wise information, forming stationary representations of words without
supervision.

Consider a sentence $\boldsymbol{x}$ passed through contextualized encoders such as ELMo and BERT with $L$ layers. Each layer of the encoders produces a vector representation for each word ${\textnormal{x}}_i$ in $\boldsymbol{x}$.
We denote by  $\boldsymbol{z}_{i,l} \in \mathbb{R}^d$ the representation given by layer $l$, a $d$-dimensional vector.
Overall,
${\textnormal{x}}_i$ receives $L$ different vectors $(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})$.
An aggregation $\phi$ maps these $L$ vectors to one final vector:
\begin{equation*}
E({\textnormal{x}}_i) = \phi(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})
\numberthis
\end{equation*}
where $E({\textnormal{x}}_i)$ is the aggregated representation of the word ${\textnormal{x}}_i$.

We study two alternatives for $\phi$:
(i) the concatenation of power means \cite{Rckl2018ConcatenatedPW} as a generalized pooling mechanism, and
(ii) a routing mechanism for aggregation~\cite{zhao:2018,zhao-etal-2019-towards}.
We relegate the routing method to appendix, as it does not yield better results than power means.

\paragraph{Power Means}
Power means
is an effective
generalization of pooling techniques for aggregating information.
It computes a non-linear average of a set of values with an exponent $p$ (Eq.~(\ref{eq:power_means})).
Following \citet{Rckl2018ConcatenatedPW}, we exploit power means to aggregate vector representations $(\boldsymbol{z}_{i,l})_{l=1}^L$ pertaining to the $i$-th word from all layers of a deep neural architecture.
Let $p\in\mathbb{R}\cup\{\pm\infty\}$, the $p$-mean of $(\boldsymbol{z}_{i,1}, \dots, \boldsymbol{z}_{i,L})$ is:
\begin{align*}

  \boldsymbol{h}_i^{(p)}=\left(\frac{\boldsymbol{z}_{i,1}^p+\cdots+\boldsymbol{z}_{i,L}^p}{L}\right)^{1/p} \in \mathbb{R}^
  \label{eq:power_means}
  \numberthis
\end{align*}
where
exponentiation is applied elementwise.
This generalized form can induce common named means such as arithmetic mean ($p=1$) and geometric mean ($p=0$). In extreme cases, a power mean reduces to the minimum value of the set when $p=-\infty$, and the maximum value when $p=+\infty$. The concatenation of $p$-mean vectors we use in this paper is denoted by:
\begin{align*}
  E(x_i) =
  \mathbf{h}_i^{(p_1)}\oplus\cdots\oplus \mathbf{h}_i^{(p_K)}
  \numberthis
\end{align*}
where $\oplus$
is vector concatenation;
$\{p_1,\ldots,p_K\}$ are exponent values, and we use $K=3$ with
$p=1,\pm\infty$
in this work.

\subsection{Summary of MoverScore Variations}
We investigate our MoverScore along four dimensions:
(i) the granularity of embeddings, i.e., the size of $n$ for $n$-grams, (ii) the choice of pretrained embedding mechanism, (iii) the fine-tuning task used for BERT\footnote{ELMo usually requires heavy layers on the top, which restricts the power of fine-tuning tasks for ELMo.} (iv) the aggregation technique ($p$-means or routing) when applicable.

\paragraph{Granularity}
We used $n = 1$ and $n = 2$ as well as full sentences ($n=$ size of the sentence).

\paragraph{Embedding Mechanism}
We obtained word embeddings from three different methods: \emph{static} embedding with word2vec as well as contextualized embedding with ELMo and BERT. If $n > 1$, $n$-gram embeddings are calculated by Eq.~(\ref{eq:n-gram-embedding}). Note that they represent sentence embeddings when $n=$ size of the sentence.

\paragraph{Fine-tuning Tasks}
Natural Language Inference (NLI) and paraphrasing pose high demands in understanding sentence meaning.
This motivated us to
fine-tune BERT representations on two NLI datasets, MultiNLI and QANLI, and one Paraphrase dataset, QQP---the largest datasets in GLUE \cite{wang:2018}.
We fine-tune BERT on each of these, yielding different contextualized embeddings for our general evaluation metrics.

\paragraph{Aggregation}
For ELMo, we aggregate
word representations given by all
three ELMo layers, using
$p$-means or routing (see the appendix).
Word representations in BERT are aggregated from the last five layers, using $p$-means or routing since the representations in the initial layers are
less suited
for use in
downstream tasks \cite{liu:2019}.

\subsection{Matching-based metrics}

\subsection{Mutiple References} All tasks except Machine Translation have multiple references associated with each system text. For Image Captioning and Dialogue Generation task, we score semantic distance between system and each available reference and obtain the highest score since the multiple references w.r.t. one system text usually have identical meanings, whereas we take the average score in Text Summarization task since the highest score is not reliable when each reference only reflects the piece meaning of source articles.
\setlength{\tabcolsep}{4.5pt}
\begin{table*}[h!]
    \setlength{\tabcolsep}{4pt}
    \footnotesize
    \centering
    \begin{tabular}{l | l ccccccc c}
    \toprule
    & & \multicolumn{8}{c}{\textbf{Direct Assessment}}\\
    Setting & Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en & Average \\

    \midrule
    \multirow{3}{*}{\metric{Baselines}}

    &\metric{METEOR++} & 0.552 & 0.538 & 0.720 & 0.563 & 0.627 & 0.626 & 0.646 & 0.610 \\

    &\metric{RUSE(*)} & 0.624 & 0.644 & 0.750 & 0.697 & 0.673 & 0.716 & 0.691 & 0.685 \\
    &\metric{BERTScore-F1} & 0.670 & 0.686 & 0.820 & 0.710 & 0.729 & 0.714 & 0.704 & 0.719 \\

    \midrule

        \multirow{4}{*}{\metric{Sent-Mover}}

    &\metric{\metric{Smd} + W2V} & 0.438 & 0.505 & 0.540 & 0.442 & 0.514 & 0.456 & 0.494 & 0.484 \\
    &\metric{\metric{Smd} + ELMO  + PMeans} & 0.569 & 0.558 & 0.732 & 0.525 & 0.581 & 0.620 & 0.584 & 0.595 \\

    &\metric{\metric{Smd} + BERT  + PMeans} & 0.607 & 0.623 & 0.770 & 0.639 & 0.667 & 0.641 & 0.619 & 0.652 \\

    &\metric{\metric{Smd} + BERT  + MNLI + PMeans} & 0.616 & 0.643 & 0.785 & 0.660 & 0.664 & 0.668 & 0.633 & 0.667 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} &0.392 & 0.463 & 0.558 & 0.463 & 0.456 & 0.485 & 0.481 & 0.471 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.579 & 0.588 & 0.753 & 0.559 & 0.617 & 0.679 & 0.645 & 0.631\\

    &\metric{Wmd-1 + BERT + PMeans} & 0.662 & 0.687 & 0.823 & 0.714 & 0.735 & 0.734 & 0.719 & 0.725\\

    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.670 & 0.708 & \textbf{0.835} & \textbf{0.746} & \textbf{0.738} & 0.762 & \textbf{0.744} & \textbf{0.743}\\

    &\metric{Wmd-2 + BERT + MNLI + PMeans} & \textbf{0.679} & \textbf{0.710} & 0.832 & 0.745 & 0.736 & \textbf{0.763} & 0.740 & \textbf{0.743}\\

	\bottomrule
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments in 7 language pairs on WMT17 dataset.

    \label{tab:wmt17-to-en} }
\vspace{-0.1in}
\end{table*}

\paragraph{Baselines}
We select multiple strong baselines for each task for comparison:
SentBLEU, METEOR++ \cite{guo:2018}, and a supervised metric RUSE for machine translation; ROUGE-1 and ROUGE-2 and a supervised metric $S^{3}_{best}$ \cite{Peyrard:2017} for text summarization; BLEU and METEOR for dialogue response generation, CIDEr, SPICE, METEOR and a supervised metric LEIC \citep{cui:2018} for image captioning. We also report
BERTScore \cite{zhang:2019} for all tasks (see \S\ref{sec:related}). Due to the page limit, we only compare with the strongest baselines, the rest can be found in the appendix.

\subsection{Machine Translation}
\label{sec:mt_results}

\paragraph{Data}
We obtain the source language sentences, their system
and reference translations from the WMT 2017
news translation shared task~\cite{Bojar:2017}.
We consider 7 language pairs: from German (de), Chinese (zh),
Czech (cs), Latvian (lv), Finnish (fi), Russian (ru),
and Turkish (tr), resp.\ to English.
Each language pair has approximately 3,000 sentences,
and each sentence has one reference translation and multiple
system translations generated by participating systems.
For each system translation, at least 15 human
assessments
are independently rated
for quality.

\paragraph{Results}
Table \ref{tab:wmt17-to-en}:
In all language pairs, the best correlation is achieved
by our word mover metrics that use a BERT pretrained
on MNLI as the embedding generator and
PMeans to aggregate the embeddings from different BERT layers, i.e.,
WMD-1/2+BERT+MNLI+PMeans.
Note that our unsupervised word mover metrics even outperforms
RUSE, a supervised metric.
We also find that our word mover metrics
outperforms the sentence mover.
We
conjecture that
important information is
lost in such a sentence representation while transforming the whole sequence of word vectors into one sentence embedding by Eq.~(\ref{eq:n-gram-embedding}).

\setlength{\tabcolsep}{3.5pt}
\begin{table*}[t]
		\small
		\centering
  		\begin{tabular}{l|l cc cc|cccc}
  		\toprule
 		&&\multicolumn{4}{c|}{TAC-2008}	&	\multicolumn{4}{c}{TAC-2009}\\
 		&&\multicolumn{2}{c}{\textbf{Responsiveness}}&\multicolumn{2}{c|}{\textbf{Pyramid}} &\multicolumn{2}{c}{\textbf{Responsiveness}} & \multicolumn{2}{c}{\textbf{Pyramid}}\\
 		Setting & Metrics	&	$r$ & $\rho$& $r$   & $\rho$ & $r$ & $\rho$ & $r$   &$\rho$\\
 		\midrule

        \multirow{4}{*}{\metric{Baselines}}

        &\metric{$S^{3}_{best}$}(*) & 0.715 & 0.595 & 0.754 & 0.652 & 0.738 & \textbf{0.595} & \textbf{0.842} & \textbf{0.731} \\
		&\metric{ROUGE-1} & 0.703 & 0.578 & 0.747 & 0.632 & 0.704 & 0.565 & 0.808 & 0.692 \\
		&\metric{ROUGE-2} & 0.695 & 0.572 & 0.718 & 0.635 & 0.727 & 0.583 & 0.803 & 0.694 \\
		&\metric{BERTScore-F1} & 0.724 & 0.594 & 0.750 & 0.649 & 0.739 & 0.580 & 0.823 & 0.703 \\
	\midrule

    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.583 & 0.469 & 0.603 & 0.488 & 0.577 & 0.465 & 0.670 & 0.560 \\

    &\metric{SMD + ELMO + PMeans} & 0.631 & 0.472 & 0.631 & 0.499 & 0.663 & 0.498 & 0.726 & 0.568 \\

    &\metric{SMD + BERT + PMeans} & 0.658 & 0.530 & 0.664 & 0.550 & 0.670 & 0.518 & 0.731 & 0.580 \\
    &\metric{SMD + BERT + MNLI + PMeans} & 0.662 & 0.525 & 0.666 & 0.552 & 0.667 & 0.506 & 0.723 & 0.563 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} & 0.669 & 0.549 & 0.665 & 0.588 & 0.698 & 0.520 & 0.740 & 0.647 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.707 & 0.554 & 0.726 & 0.601 & 0.736 & 0.553 & 0.813 & 0.672 \\
    &\metric{Wmd-1 + BERT + PMeans} & 0.729 & 0.595 & 0.755 & 0.660 & 0.742 & 0.581 & 0.825 & 0.690 \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & \textbf{0.736} & \textbf{0.604} & \textbf{0.760} & \textbf{0.672} & \textbf{0.754} & 0.594 & 0.831 & 0.701 \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.734 & 0.601 & 0.752 & 0.663 & 0.753 & 0.586 & 0.825 & 0.694 \\

		\bottomrule
		\end{tabular}
 		\caption{Pearson $r$ and Spearman $\rho$ correlations with summary-level human judgments on TAC 2008 and 2009.
 		}
 		\label{tab:tac0809}
\vspace{-0.08in}
	\end{table*}

\subsection{Text Summarization}
\label{sec:summ_results}

We use two summarization datasets from the Text Analysis Conference
(TAC)\footnote{\url{http://tac.nist.gov}}:
TAC-2008 and TAC-2009, which
contain 48 and 44 \emph{clusters}, respectively.
Each cluster includes 10 news articles (on the same topic),
four reference summaries,
and 57 (in TAC-2008) or 55 (in TAC-2009) system
summaries generated by the participating systems.
Each summary (either reference or system) has fewer than 100 words,
and receives two human judgment scores:
the \emph{Pyramid} score~\cite{nenkova-passonneau:2004:HLTNAACL}
and the \emph{Responsiveness} score.
Pyramid measures how many important semantic content units
in the reference summaries are covered by the system summary,
while Responsiveness measures
how well a summary responds to the overall quality combining both content and linguistic quality.

\paragraph{Results}
Tables \ref{tab:tac0809}:
We observe that lexical metrics like ROUGE correlate above-moderate on TAC 2008 and 2009 datasets. In contrast, these metrics perform poorly on other tasks like Dialogue Generation \cite{novikova-etal-2017-need} and Image Captioning \cite{AndersonFJG16}.
Apparently, strict matches on surface forms seems reasonable for \emph{extractive} summarization datasets.
However, we still
see that our word mover metrics, i.e., WMD-1+BERT+MNLI+PMeans,
perform better than or
come close to
even the
supervised metric $S^{3}_{best}$.

\setlength{\tabcolsep}{8.3pt}
\begin{table*}
	\small
	\centering
	\begin{tabular}{l|l|c c c|c c c}
		\toprule
		&& \multicolumn{3}{c}{BAGEL} & \multicolumn{3}{c}{SFHOTEL} \\
        Setting&Metrics & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} \\
		\midrule
		\noalign{\smallskip}

        \multirow{4}{*}{\metric{Baselines}}
        &\metric{BLEU-1} & 0.225 & 0.141 & 0.113 & 0.107 & 0.175 & 0.069 \\
        &\metric{BLEU-2} & 0.211 & 0.152 & 0.115 & 0.097 & 0.174 & 0.071 \\

        &\metric{METEOR} & 0.251 & 0.127 & 0.116 & 0.111 & 0.148 & 0.082 \\
        &\metric{BERTScore-F1} & 0.267 & 0.210 & \textbf{0.178} & 0.163 & 0.193 & 0.118 \\

    \midrule

    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.024 & 0.074 & 0.078 & 0.022 & 0.025 & 0.011 \\
    &\metric{SMD + ELMO + PMeans} & 0.251 & 0.171 & 0.147 & 0.130 & 0.176 & 0.096 \\

    &\metric{SMD + BERT + PMeans} & 0.290 & 0.163 & 0.121 & 0.192 & 0.223 & 0.134 \\

    &\metric{SMD + BERT + MNLI + PMeans} & 0.280 & 0.149 & 0.120 & 0.205 & 0.239 & 0.147 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} & 0.222 & 0.079 & 0.123 & 0.074 & 0.095 & 0.021 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.261 & 0.163 & 0.148 & 0.147 & 0.215 & 0.136  \\
    &\metric{Wmd-1 + BERT + PMeans} & \textbf{0.298} & \textbf{0.212} & 0.163 & 0.203 & 0.261 & 0.182  \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.285 & 0.195 & 0.158 & \textbf{0.207} & \textbf{0.270} & \textbf{0.183}  \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.284 & 0.194 & 0.156 & 0.204 & 0.270 & 0.182  \\

		\bottomrule
	\end{tabular}
	\caption{Spearman correlation with utterance-level human judgments for BAGEL and SFHOTEL datasets.
	}
	\label{tab:bag_sfh}
\vspace{-0.1in}
\end{table*}
\setlength{\tabcolsep}{1.4pt}
\subsection{Data-to-text Generation}
\label{sec:gen_results}
We use two task-oriented dialogue datasets: BAGEL \cite{mairesse:2010} and SFHOTEL \cite{wen:2015}, which contains 202 and 398 instances of Meaning Representation (MR). Each MR instance includes multiple references, and roughly two system utterances generated by different neural systems. Each system utterance receives three human judgment scores: \emph{informativeness}, \emph{naturalness} and \emph{quality} score \cite{novikova-etal-2017-need}. Informativeness
measures
how much information a system utterance
provides with respect to an MR. Naturalness measures how likely a system utterance is generated by native speakers. Quality %
measures
how well a system utterance
captures fluency and grammar.
\paragraph{Results}

Tables \ref{tab:bag_sfh}:
Interestingly,
no metric produces an even moderate correlation with human judgments, including our own. We speculate that current contextualizers are poor at representing named entities like hotels and place names as well as numbers appearing in system and reference texts.
However,
best correlation is still achieved by our word mover metrics combining contextualized representations.

\subsection{Image Captioning}
\label{sec:cap_results}
We use a
popular
image captioning dataset: MS-COCO \cite{lin:2014:coco}, which contains 5,000 images. Each image includes roughly five reference captions, and 12 system captions generated by the participating systems from 2015 COCO Captioning Challenge. For the system-level human correlation, each system receives five human judgment scores: M1, M2, M3, M4, M5 \citep{AndersonFJG16}. The M1 and M2 scores
measure overall quality of the captions while M3, M4 and M5 scores
measure correctness, detailedness and saliency of the captions.
Following \citet{cui:2018}, we compare the Pearson correlation with two system-level scores: M1 and M2, since we focus on studying metrics
for the overall quality of the captions, leaving metrics understanding captions in different aspects (correctness, detailedness and saliency) to future work.

\paragraph{Results}
Table \ref{tab:caption}:
Word mover metrics outperform all baselines except for the supervised metric LEIC, which  uses more information by considering \emph{both} images and texts.
\setlength{\tabcolsep}{3.7pt}
\begin{table}[t!]
    \small
    \centering
    \begin{tabular}{l|l | c c}
    \toprule
    Setting & Metric & M1 & M2 \\
    \midrule

    \multirow{5}{*}{\textsc{Baselines}}
    &\metric{LEIC(*)} & \textbf{0.939} & \textbf{0.949} \\
    &\metric{METEOR} & 0.606 & 0.594 \\

    &\metric{SPICE} & 0.759 & 0.750 \\
    &\metric{BERTScore-Recall} & 0.809 & 0.749\\

    \midrule
    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.683 & 0.668\\
    &\metric{SMD + ELMO + P} & 0.709 & 0.712\\
    &\metric{SMD + BERT + P} & 0.723 & 0.747\\
    &\metric{SMD + BERT + M + P} & 0.789 & 0.784\\
    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}

    &\metric{Wmd-1 + W2V} & 0.728 & 0.764\\
    &\metric{Wmd-1 + ELMO + P} & 0.753 & 0.775\\
    &\metric{Wmd-1 + BERT + P} & 0.780 & 0.790\\
    &\metric{Wmd-1 + BERT + M + P} & \textbf{0.813} & \textbf{0.810}\\
    &\metric{Wmd-2 + BERT + M + P} & 0.812 & 0.808\\

    \bottomrule
    \end{tabular}
    \caption{Pearson correlation with system-level human judgments on MSCOCO dataset. 'M' and 'P' are short names.}
    \label{tab:caption}
\vspace{-0.05in}
\end{table}

\setlength{\tabcolsep}{5pt}
\begin{table}[h!]
    \footnotesize
    \centering
    \begin{tabular}{l|cccc}
    \toprule

    Metrics & cs-en & de-en & fi-en & lv-en \\

    \midrule

    \metric{RUSE} & 0.624 & 0.644 & 0.750 & 0.697 \\
    \midrule

    \metric{\metric{Hmd-F1} + BERT} & 0.655 & 0.681 & 0.821 & 0.712 \\
    \metric{\metric{Hmd-Recall} + BERT} & 0.651 & 0.658 & 0.788 & 0.681\\
    \metric{\metric{Hmd-Prec} + BERT} & 0.624 & 0.669 & 0.817 & 0.707 \\
    \midrule
    \metric{Wmd-unigram + BERT} & 0.651 & 0.686 & \textbf{0.823} & 0.710 \\
    \metric{Wmd-bigram + BERT} & \textbf{0.665} & \textbf{0.688} & 0.821 & \textbf{0.712} \\

    \hline
    \end{tabular}

    \caption{Comparison on hard and soft alignments.}
    \label{tab:wmt17-ablation}
\vspace{-0.05in}
\end{table}

\subsection{Further Analysis}
\noindent\textbf{Hard and Soft Alignments}\quad
BERTScore is the harmonic mean of BERTScore-Precision and BERTScore-Recall, where both two can be
decomposed
as a combination of ``Hard Mover Distance'' (HMD) and BERT (see Prop.~\ref{prop:main}).

We use the representations in the 9-th BERT layer for fair comparison of
BERTScore and MoverScore
and show results on the machine translation task in Table \ref{tab:wmt17-ablation}.
MoverScore outperforms both asymmetric HMD factors, while if they are combined via harmonic mean, BERTScore is on par with MoverScore. We conjecture that BERT softens hard alignments of BERTScore as contextualized embeddings encode information about the whole sentence into each word vector.
We also observe that \metric{Wmd-bigrams} slightly
outperforms \metric{Wmd-unigrams} on 3 out of 4 language pairs.

\vspace{0.1in}
\noindent\textbf{Distribution of Scores}\quad
In Figure \ref{fig:score_dist}, we take a closer look at sentence-level correlation in MT.
Results reveal that the lexical metric \metric{SentBLEU} can correctly assign lower scores to system translations of low quality, while it struggles in judging system translations of high quality by assigning them lower scores. Our finding agrees with the observations found in \citet{chaganty:2018,novikova-etal-2017-need}: lexical metrics correlate better with human judgments on texts of low quality than high quality.
\citet{peyrard-2019-studying} further show that lexical metrics
cannot be trusted because they strongly disagree on high-scoring system outputs.
Importantly, we observe that our word mover metric combining BERT can clearly distinguish texts of two polar qualities.

\begin{figure}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/score-dist-SentBLEU.pdf}}
\end{minipage}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/score-dist-BERT.pdf}}
\end{minipage}
 \caption{
 Score distribution in German-to-English pair.
 }
 \label{fig:score_dist}
\vspace{-0.1in}
\end{figure}

\begin{figure}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/corrplot_de_en.pdf}}
\end{minipage}
\begin{minipage}{0.48\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/corrplot_zh_en.pdf}}
\end{minipage}
 \caption{
 Correlation in similar language (de-en) and distant language (zh-en) pair, where bordered area shows correlations between human assessment and metrics, the rest shows inter-correlations across metrics and DA is direct assessment rated by language experts.
 }
 \label{fig:color_example}
\vspace{-0.15in}
\end{figure}

\vspace{0.1in}
\noindent\textbf{Correlation Analysis}\quad
In Figure \ref{fig:color_example}, we observe existing metrics for MT evaluation attaining medium  correlations (0.4-0.5) with human judgments but high inter-correlations between themselves.
In contrast,
our metrics
can attain high correlations (0.6-0.7) with human judgments, performing robust across different language pairs.
We believe that our improvements come from clearly distinguishing translations that fall on two extremes.

\vspace{0.1in}
\noindent\textbf{Impact of Fine-tuning Tasks}\quad
Figure \ref{fig:task-transfer-analysis} compares Pearson correlations with our word mover metrics combining BERT fine-tuned on three different tasks. We observe that fine-tuning on closely related tasks improves correlations, especially fine-tuning on MNLI leads to an impressive improvement by 1.8 points on average.
\begin{figure}[t]
\centering
\includegraphics[width=.6\linewidth]{figures/task-transfer-wmt17.pdf} %
\caption{Correlation is averaged over 7 language pairs.}
 \label{fig:task-transfer-analysis}
\vspace{-0.05in}
\end{figure}

\input{discussion.tex}

\subsection{Proof of Prop.~\ref{prop:main}}

In this section, we prove Prop.~\ref{prop:main} in the paper about viewing BERTScore (precision/recall) as a (non-optimized) Mover Distance.

As a reminder, the WMD formulation is:
\begin{align*}
& \text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := \min_{{\bm{F}}\in\mathbb{R}^{|\boldsymbol{x}^n|\times |\boldsymbol{y}^n|}}\sum_{i,j} {\bm{C}}_{ij} \cdot {\bm{F}}_{ij}\\
&\text{s.t. } \boldsymbol{1}^{\intercal}{\bm{F}}^{\intercal}\boldsymbol{1}=1,\;\;\boldsymbol{1}^{\intercal}{\bm{F}}\boldsymbol{1}=1.
\end{align*}
where ${\bm{F}}^{\intercal}\boldsymbol{1}=\boldsymbol{f}_x^n$ and ${\bm{F}}\boldsymbol{1}=\boldsymbol{f}_y^n$. Here, $\boldsymbol{f}_x^n$ and $\boldsymbol{f}_y^n$ denote vectors of weights for each $n$-gram of $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$.

BERTScore is defined as:
\begin{align}
R_\bert  &= \frac{\sum_{y_i^1 \in \boldsymbol{y}^1} {\rm idf}(y_i^1)  \max_{x_j^1 \in \boldsymbol{x}^1} E(x_j^1)^\intercal E(y_i^1)}{\sum_{y_i^1\in \boldsymbol{y}^1} {\rm idf}(y_i^1)} \nonumber \\
P_\bert  &= \frac{\sum_{x_j^1 \in \boldsymbol{x}^1} {\rm idf}(x_j^1)  \max_{y_i^1 \in \boldsymbol{y}^1} E(y_i^1)^\intercal E(x_j^1)}{\sum_{x_j^1\in \boldsymbol{x}^1} {\rm idf}(x_j^1)} \nonumber \\
F_\bert &= 2\frac{P_\bert \cdot R_\bert }{ P_\bert + R_\bert }\;\;. \nonumber
\end{align}
Then, $R_\bert$ can be formulated in a ``quasi'' WMD form:

\begin{align*}
& R_\bert(\boldsymbol{x}^1,\boldsymbol{y}^1) := \sum_{i,j} {\bm{C}}_{ij} \cdot {\bm{F}}_{ij}\\
&{\bm{F}}_{ij}=\begin{cases}
      \frac{1}{M} &

      \text{if } x_j=\argmax_{\hat{x}_j^1\in \boldsymbol{x}^1}E(y_i^1)^\intercal E(\hat{x}_j^1)\\
      0 & \text{otherwise}
\end{cases}\\
&{\bm{C}}_{ij}=\begin{cases}
      \frac{M}{Z}{\rm idf}(y_i^1) E(x_j^1)^\intercal E(y_i^1) & \text{if } x_j=\argmax_{\hat{x}_j^1\in \boldsymbol{x}^1}E(y_i^1)^\intercal E(\hat{x}_j^1)\\
      0 & \text{otherwise}
\end{cases}\\
\end{align*}
where $Z=\sum_{y_i^1 \in \boldsymbol{y}^1}{\rm idf}(y_i^1)$ and $M$ is the size of $n$-grams in $\boldsymbol{x}^1$. Similarly, we can have $P_\bert$ in a quasi WMD form (omitted). Then, $F_\bert$ can be formulated as harmonic-mean of two WMD forms of $P_\bert$ and $R_\bert$.

\subsection{Routing}
In this section, we study the aggregation function $\phi$ with a routing scheme, which has achieved good results in other NLP tasks \cite{zhao:2018, zhao-etal-2019-towards}.
Specifically, we introduce a nonparametric clustering with Kernel Density Estimation (KDE) for routing since KDE bridges a family of kernel functions with underlying empirical distributions, which often leads to computational efficiency \cite{zhangfast}, defined as:
\begin{align*}
&\min_{\boldsymbol{v},\gamma}
f(\boldsymbol{z})
=\sum_{i=1}^L \sum_{j=1}^T\gamma_{ij}k(d(\boldsymbol{v}_j - \boldsymbol{z}_{i,j}))\\
& \textit{s.t.}\quad\forall{i,j}:\gamma_{ij}>0,\sum\limits_{j=1}^{L}\gamma_{ij}=1.
\end{align*}
where $d(\cdot)$ is a distance function, $\gamma_{ij}$ denotes the  underlying closeness between the aggregated vector $\boldsymbol{v}_j$ and vector $\boldsymbol{z}_i$ in the $i$-th layer, and $k$ is a kernel function. Some instantiations of $k(\cdot)$ \cite{wand:1994} are:
\begin{align*}
\mathit{Gaussian}: k(x)\triangleq\exp{(-\frac{x}{2})},\;\;
\mathit{Epanechnikov}: k(x)\triangleq\left\{\begin{aligned}1-x && x\in{[0,1)}\\ 0 && x\ge{1}.\end{aligned}\right.
\end{align*}
One typical solution for KDE clustering to minimize $f(\boldsymbol{z})$ is taking Mean Shift \cite{comaniciu:2002}, defined as:
\begin{align*}
	\nabla f(\boldsymbol{z})=\sum_{i,j} \gamma_{ij} k'(d(\boldsymbol{v}_j, \boldsymbol{z}_{i,j}))\frac{\partial{d(\boldsymbol{v}_j,  \boldsymbol{z}_{i,j})}}{\partial{\boldsymbol{v}}} \nonumber
\end{align*}
Firstly, $\boldsymbol{v}_j^{\tau+1}$ can be updated while $\gamma_{ij}^{\tau+1}$ is fixed:
\begin{align*}
	\boldsymbol{v}_j^{\tau+1} = \frac{\sum_{i} \gamma_{ij}^\tau k'(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j})) \boldsymbol{z}_{i,j}}{\sum_{i,j} k'(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j}))}
\end{align*}
Intuitively, $\boldsymbol{v}_j$ can be explained as a final aggregated vector from $L$ contextualized layers.
Then, we adopt SGD to update $\gamma_{ij}^{\tau+1}$:
\begin{align*}
	\gamma_{ij}^{\tau+1} = \gamma_{ij}^\tau + \alpha \cdot k(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j}))
\end{align*}
where $\alpha$ is a hyperparameter to control step size. The routing process is summarized in Algorithm \ref{alg:1}.

\begin{algorithm}
\caption{Aggregation by Routing}\label{alg:1}
\begin{algorithmic}[1]
\small
\State \textbf{procedure} ROUTING($\boldsymbol{z}_{ij}$, $\ell$)
\State Initialize $\forall i,j: \gamma_{ij}=0$
\While {true}
  \ForAll{representation $i$ and $j$ in layer $\ell$ and  $\ell+1$}

    $\gamma_{ij} \gets \mathit{softmax}(\gamma_{ij})$
  \EndFor

  \ForAll{representation $j$ in layer $\ell+1$}
    \State $\boldsymbol{v}_j \gets \sum_{i} \gamma_{ij} k'(\boldsymbol{v}_j, \boldsymbol{z}_i) \boldsymbol{z}_i / \sum_{i} k'(\boldsymbol{v}_i, \boldsymbol{z}_i)$
  \EndFor

  \ForAll{representation $i$ and $j$ in layer $\ell$ and  $\ell+1$} $\gamma_{ij}\gets \gamma_{ij} + \alpha \cdot k(\boldsymbol{v}_j, \boldsymbol{z}_i)$

  \EndFor

  \State $\rm{loss} \gets \log(\sum_{i,j}\gamma_{ij}k(\boldsymbol{v}_j, \boldsymbol{z}_i))$
  \If{$|\rm{loss}- \rm{preloss}| < \epsilon$}
      \State \textbf{break}
  \Else
      \State $\rm{preloss} \gets \rm{loss}$
  \EndIf
\EndWhile
\State \textbf{return} $v_j$
\end{algorithmic}
\end{algorithm}

\paragraph{Best Layer and Layer-wise Consolidation}
Table \ref{tab:wmt17-aggregation} compares our word mover based metric combining BERT representations on different layers with stronger BERT representations consolidated from these layers (using $p$-means and routing). We often see that which layer has best performance is task-dependent, and our word mover based metrics (WMD) with $p$-means or routing schema come close to the oracle performance obtained from the best layers.

\setlength{\tabcolsep}{8.8pt}
\begin{table*}[h!]
    \small
    \centering
    \begin{tabular}{l ccccccc}
    \toprule
    & \multicolumn{7}{c}{\textbf{Direct Assessment}}\\
    Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en \\

    \midrule

    \metric{Wmd-1 + BERT + Layer 8} & .6361 & .6755 & .8134 & .7033 & .7273 & .7233 & .7175 \\
    \metric{Wmd-1 + BERT + Layer 9} & .6510 & .6865 & .8240 & .7107 & .7291 & \textbf{.7357} & \textbf{.7195} \\
    \metric{Wmd-1 + BERT + Layer 10} & .6605 & \textbf{.6948} & \textbf{.8231} & .7158 & \textbf{.7363} & .7317 & .7168 \\
    \metric{Wmd-1 + BERT + Layer 11} & \textbf{.6695} & .6845 & .8192 & .7048 & .7315 & .7276 & .7058 \\
    \metric{Wmd-1 + BERT + Layer 12} & .6677 & .6825 & .8194 & \textbf{.7188} & .7326 & .7291 & .7064 \\
    \midrule

    \metric{Wmd-1 + BERT + Routing} & .6618 & .6897 & .8225 & .7122 & .7334 & .7301 & .7182 \\
    \metric{Wmd-1 + BERT + PMeans} & .6623 & .6873 & .8234 & .7139 & .7350 & .7339 & .7192 \\

	\hline
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations.}
    \label{tab:wmt17-aggregation}
\end{table*}

\paragraph{Experiments}
Table ~\ref{tab:wmt17-all}, \ref{tab:tac-all} and \ref{tab:dialogue-all} show correlations between metrics (all baseline metrics and word mover based metrics) and human judgments on machine translation, text summarization and dialogue response generation, respectively. We find that word mover based metrics combining BERT fine-tuned on MNLI
have highest correlations with humans,
outperforming
all of the unsupervised metrics and even
supervised metrics like $\metric{RUSE}$ and $\metric{$S^{3}_{full}$}$.
Routing and $p$-means
perform roughly equally well.

\setlength{\tabcolsep}{5pt}
\begin{table}[h!]
    \footnotesize
    \centering
    \begin{tabular}{c|l|ccccccc c}
    \toprule
    & & \multicolumn{8}{c}{\textbf{Direct Assessment}}\\
    Setting & Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en & Average \\

    \midrule
    \multirow{5}{*}{\metric{Baselines}}
    & \metric{BLEND} & 0.594 & 0.571 & 0.733 & 0.594 & 0.622 & 0.671 & 0.661 & 0.635 \\
    & \metric{RUSE} & 0.624 & 0.644 & 0.750 & 0.697 & 0.673 & 0.716 & 0.691 & 0.685 \\

    & \metric{SentBLEU} & 0.435 & 0.432 & 0.571 & 0.393 & 0.484 & 0.538 & 0.512 & 0.481 \\
    & \metric{chrF++} & 0.523 & 0.534 & 0.678 & 0.520 & 0.588 & 0.614 & 0.593 & 0.579 \\
    & \metric{METEOR++} & 0.552 & 0.538 & 0.720 & 0.563 & 0.627 & 0.626 & 0.646 & 0.610 \\
    & \metric{BERTScore-F1} & 0.670 & 0.686 & 0.820 & 0.710 & 0.729 & 0.714 & 0.704 & 0.719 \\
    \midrule

    \multirow{7}{*}{\metric{Word-Mover}}

    & \metric{Wmd-1 + W2V} &0.392 & 0.463 & 0.558 & 0.463 & 0.456 & 0.485 & 0.481 & 0.471 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.658 & 0.689 & 0.823 & 0.712 & 0.733 & 0.730 & 0.718 & 0.723 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.665 & 0.705 & 0.834 & 0.744 & 0.735 & 0.752 & 0.736 & 0.739 \\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.676 & 0.706 & 0.831 & 0.743 & 0.734 & 0.755 & 0.732 & 0.740 \\

    &\metric{Wmd-1 + BERT + PMeans} & 0.662 & 0.687 & 0.823 & 0.714 & 0.735 & 0.734 & 0.719 & 0.725\\

    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.670 & 0.708 & \textbf{0.835} & \textbf{0.746} & \textbf{0.738} & 0.762 & \textbf{0.744} & \textbf{0.743}\\

    &\metric{Wmd-2 + BERT + MNLI + PMeans} & \textbf{0.679} & \textbf{0.710} & 0.832 & 0.745 & 0.736 & \textbf{0.763} & 0.740 & \textbf{0.743}\\

	\hline
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations.}
    \label{tab:wmt17-all}
\end{table}

\setlength{\tabcolsep}{4.5pt}
\begin{table}
		\small
		\centering
  		\begin{tabular}{c|l |cc|cc|cc|cc}
  		\toprule
 		&&\multicolumn{4}{c|}{TAC-2008}	&	\multicolumn{4}{c}{TAC-2009}\\
 		&&\multicolumn{2}{c|}{\textbf{Responsiveness}}&\multicolumn{2}{c|}{\textbf{Pyramid}} &\multicolumn{2}{c|}{\textbf{Responsiveness}} & \multicolumn{2}{c}{\textbf{Pyramid}}\\
 		Setting & Metrics	&	$r$ & $\rho$& $r$   & $\rho$ & $r$ & $\rho$ & $r$   &$\rho$\\
 		\midrule

        \multirow{13}{*}{\metric{Baselines}}
        &\metric{$S^{3}_{full}$} & 0.696 & 0.558 & 0.753 & 0.652 & 0.731 & 0.552 & 0.838 & 0.724 \\
		&\metric{$S^{3}_{best}$} & 0.715 & 0.595 & 0.754 & 0.652 & 0.738 & \textbf{0.595} & \textbf{0.842} & \textbf{0.731} \\

		&\metric{TF$*$IDF-1}	& 0.176 & 0.224 & 0.183 & 0.237 & 0.187 & 0.222 & 0.242 & 0.284\\
		&\metric{TF$*$IDF-2}	& 0.047 & 0.154 & 0.049 & 0.182 & 0.047 & 0.167 & 0.097 & 0.233\\
		&\metric{ROUGE-1} & 0.703 & 0.578 & 0.747 & 0.632 & 0.704 & 0.565 & 0.808 & 0.692 \\
		&\metric{ROUGE-2} & 0.695 & 0.572 & 0.718 & 0.635 & 0.727 & 0.583 & 0.803 & 0.694 \\
		&\metric{ROUGE-1-WE}	& 0.571 & 0.450 & 0.579 & 0.458 & 0.586 & 0.437 & 0.653 & 0.516 \\
		&\metric{ROUGE-2-WE}	& 0.566 & 0.397 & 0.556 & 0.388 & 0.607 & 0.413 & 0.671 & 0.481 \\
		&\metric{ROUGE-L} & 0.681 & 0.520 & 0.702 & 0.568 & 0.730 & 0.563 & 0.779 & 0.652 \\
		&\metric{Frame-1} & 0.658 & 0.508 & 0.686 & 0.529 & 0.678 & 0.527 & 0.762 & 0.628 \\
		&\metric{Frame-2} & 0.676 & 0.519 & 0.691 & 0.556 & 0.715 & 0.555 & 0.781 & 0.648 \\
        & \metric{BERTScore-F1} & 0.724 & 0.594 & 0.750 & 0.649 & 0.739 & 0.580 & 0.823 & 0.703 \\
		\midrule

    \multirow{7}{*}{\metric{Word-Mover}}

    & \metric{Wmd-1 + W2V} & 0.669 & 0.559 & 0.665 & 0.611 & 0.698 & 0.520 & 0.740 & 0.647 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.729 & 0.601 & 0.763 & 0.675 & 0.740 & 0.580 & 0.831 & 0.700 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.734 & \textbf{0.609} & \textbf{0.768} & \textbf{0.686} & 0.747 & 0.589 & 0.837 & 0.711 \\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.731 & 0.593 & 0.755 & 0.666 & 0.753 & 0.583 & 0.827 & 0.698 \\
    &\metric{Wmd-1 + BERT + PMeans} & 0.729 & 0.595 & 0.755 & 0.660 & 0.742 & 0.581 & 0.825 & 0.690 \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & \textbf{0.736} & 0.604 & 0.760 & 0.672 & \textbf{0.754} & 0.594 & 0.831 & 0.701 \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.734 & 0.601 & 0.752 & 0.663 & 0.753 & 0.586 & 0.825 & 0.694 \\

		\hline
		\end{tabular}
 		\caption{Correlation of automatic metrics with summary-level human judgments for TAC-2008 and TAC-2009.}
 		\label{tab:tac-all}
	\end{table}
\vspace{-0.1in}

\setlength{\tabcolsep}{8pt}
\begin{table}
    \small
	\centering

	\begin{tabular}{c | l|c c c|c c c}
		\hline\noalign{\smallskip}
		&& \multicolumn{3}{c}{BAGEL} & \multicolumn{3}{c}{SFHOTEL} \\
        Setting & Metrics & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} \\
		\hline\noalign{\smallskip}

        \multirow{8}{*}{\metric{Baselines}}
        &\metric{BLEU-1} & 0.225 & 0.141 & 0.113 & 0.107 & 0.175 & 0.069 \\
        &\metric{BLEU-2} & 0.211 & 0.152 & 0.115 & 0.097 & 0.174 & 0.071 \\
        &\metric{BLEU-3} & 0.191 & 0.150 & 0.109 & 0.089 & 0.161 & 0.070 \\
        &\metric{BLEU-4} & 0.175 & 0.141 & 0.101 & 0.084 & 0.104 & 0.056 \\
        &\metric{ROUGE-L} & 0.202 & 0.134 & 0.111 & 0.092 & 0.147 & 0.062 \\
        &\metric{NIST} & 0.207 & 0.089 & 0.056 & 0.072 & 0.125 & 0.061 \\
        &\metric{CIDEr} & 0.205 & 0.162 & 0.119 & 0.095 & 0.155 & 0.052 \\
        &\metric{METEOR} & 0.251 & 0.127 & 0.116 & 0.111 & 0.148 & 0.082 \\
        & \metric{BERTScore-F1} & 0.267 & 0.210 & \textbf{0.178} & 0.163 & 0.193 & 0.118 \\

    \midrule
    \multirow{7}{*}{\metric{Word-Mover}}
    & \metric{Wmd-1 + W2V} & 0.222 & 0.079 & 0.123 & 0.074 & 0.095 & 0.021 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.294 & 0.209 & 0.156 & 0.208 & 0.256 & 0.178 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.278 & 0.180 & 0.144 & \textbf{0.211} & 0.252 & 0.175\\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.279 & 0.182 & 0.147 & 0.204 & 0.252 & 0.172 \\
    &\metric{Wmd-1 + BERT + PMeans} & \textbf{0.298} & \textbf{0.212} & 0.163 & 0.203 & 0.261 & 0.182  \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.285 & 0.195 & 0.158 & 0.207 & \textbf{0.270} & \textbf{0.183}  \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.284 & 0.194 & 0.156 & 0.204 & \textbf{0.270} & 0.182  \\

		\hline
	\end{tabular}
	\caption{Spearman correlation with utterance-level human judgments for BAGEL and SFHOTEL datasets.}
	\label{tab:dialogue-all}
\end{table}
\setlength{\tabcolsep}{1.4pt}

\subsection{Matching-based metrics}

\subsection{Mutiple References} All tasks except Machine Translation have multiple references associated with each system text. For Image Captioning and Dialogue Generation task, we score semantic distance between system and each available reference and obtain the highest score since the multiple references w.r.t. one system text usually have identical meanings, whereas we take the average score in Text Summarization task since the highest score is not reliable when each reference only reflects the piece meaning of source articles.
\setlength{\tabcolsep}{4.5pt}
\begin{table*}[h!]
    \setlength{\tabcolsep}{4pt}
    \footnotesize
    \centering
    \begin{tabular}{l | l ccccccc c}
    \toprule
    & & \multicolumn{8}{c}{\textbf{Direct Assessment}}\\
    Setting & Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en & Average \\

    \midrule
    \multirow{3}{*}{\metric{Baselines}}

    &\metric{METEOR++} & 0.552 & 0.538 & 0.720 & 0.563 & 0.627 & 0.626 & 0.646 & 0.610 \\

    &\metric{RUSE(*)} & 0.624 & 0.644 & 0.750 & 0.697 & 0.673 & 0.716 & 0.691 & 0.685 \\
    &\metric{BERTScore-F1} & 0.670 & 0.686 & 0.820 & 0.710 & 0.729 & 0.714 & 0.704 & 0.719 \\

    \midrule

        \multirow{4}{*}{\metric{Sent-Mover}}

    &\metric{\metric{Smd} + W2V} & 0.438 & 0.505 & 0.540 & 0.442 & 0.514 & 0.456 & 0.494 & 0.484 \\
    &\metric{\metric{Smd} + ELMO  + PMeans} & 0.569 & 0.558 & 0.732 & 0.525 & 0.581 & 0.620 & 0.584 & 0.595 \\

    &\metric{\metric{Smd} + BERT  + PMeans} & 0.607 & 0.623 & 0.770 & 0.639 & 0.667 & 0.641 & 0.619 & 0.652 \\

    &\metric{\metric{Smd} + BERT  + MNLI + PMeans} & 0.616 & 0.643 & 0.785 & 0.660 & 0.664 & 0.668 & 0.633 & 0.667 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} &0.392 & 0.463 & 0.558 & 0.463 & 0.456 & 0.485 & 0.481 & 0.471 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.579 & 0.588 & 0.753 & 0.559 & 0.617 & 0.679 & 0.645 & 0.631\\

    &\metric{Wmd-1 + BERT + PMeans} & 0.662 & 0.687 & 0.823 & 0.714 & 0.735 & 0.734 & 0.719 & 0.725\\

    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.670 & 0.708 & \textbf{0.835} & \textbf{0.746} & \textbf{0.738} & 0.762 & \textbf{0.744} & \textbf{0.743}\\

    &\metric{Wmd-2 + BERT + MNLI + PMeans} & \textbf{0.679} & \textbf{0.710} & 0.832 & 0.745 & 0.736 & \textbf{0.763} & 0.740 & \textbf{0.743}\\

	\bottomrule
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments in 7 language pairs on WMT17 dataset.

    \label{tab:wmt17-to-en} }
\vspace{-0.1in}
\end{table*}

\paragraph{Baselines}
We select multiple strong baselines for each task for comparison:
SentBLEU, METEOR++ \cite{guo:2018}, and a supervised metric RUSE for machine translation; ROUGE-1 and ROUGE-2 and a supervised metric $S^{3}_{best}$ \cite{Peyrard:2017} for text summarization; BLEU and METEOR for dialogue response generation, CIDEr, SPICE, METEOR and a supervised metric LEIC \citep{cui:2018} for image captioning. We also report
BERTScore \cite{zhang:2019} for all tasks (see \S\ref{sec:related}). Due to the page limit, we only compare with the strongest baselines, the rest can be found in the appendix.

\subsection{Machine Translation}
\label{sec:mt_results}

\paragraph{Data}
We obtain the source language sentences, their system
and reference translations from the WMT 2017
news translation shared task~\cite{Bojar:2017}.
We consider 7 language pairs: from German (de), Chinese (zh),
Czech (cs), Latvian (lv), Finnish (fi), Russian (ru),
and Turkish (tr), resp.\ to English.
Each language pair has approximately 3,000 sentences,
and each sentence has one reference translation and multiple
system translations generated by participating systems.
For each system translation, at least 15 human
assessments
are independently rated
for quality.

\paragraph{Results}
Table \ref{tab:wmt17-to-en}:
In all language pairs, the best correlation is achieved
by our word mover metrics that use a BERT pretrained
on MNLI as the embedding generator and
PMeans to aggregate the embeddings from different BERT layers, i.e.,
WMD-1/2+BERT+MNLI+PMeans.
Note that our unsupervised word mover metrics even outperforms
RUSE, a supervised metric.
We also find that our word mover metrics
outperforms the sentence mover.
We
conjecture that
important information is
lost in such a sentence representation while transforming the whole sequence of word vectors into one sentence embedding by Eq.~(\ref{eq:n-gram-embedding}).

\setlength{\tabcolsep}{3.5pt}
\begin{table*}[t]
		\small
		\centering
  		\begin{tabular}{l|l cc cc|cccc}
  		\toprule
 		&&\multicolumn{4}{c|}{TAC-2008}	&	\multicolumn{4}{c}{TAC-2009}\\
 		&&\multicolumn{2}{c}{\textbf{Responsiveness}}&\multicolumn{2}{c|}{\textbf{Pyramid}} &\multicolumn{2}{c}{\textbf{Responsiveness}} & \multicolumn{2}{c}{\textbf{Pyramid}}\\
 		Setting & Metrics	&	$r$ & $\rho$& $r$   & $\rho$ & $r$ & $\rho$ & $r$   &$\rho$\\
 		\midrule

        \multirow{4}{*}{\metric{Baselines}}

        &\metric{$S^{3}_{best}$}(*) & 0.715 & 0.595 & 0.754 & 0.652 & 0.738 & \textbf{0.595} & \textbf{0.842} & \textbf{0.731} \\
		&\metric{ROUGE-1} & 0.703 & 0.578 & 0.747 & 0.632 & 0.704 & 0.565 & 0.808 & 0.692 \\
		&\metric{ROUGE-2} & 0.695 & 0.572 & 0.718 & 0.635 & 0.727 & 0.583 & 0.803 & 0.694 \\
		&\metric{BERTScore-F1} & 0.724 & 0.594 & 0.750 & 0.649 & 0.739 & 0.580 & 0.823 & 0.703 \\
	\midrule

    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.583 & 0.469 & 0.603 & 0.488 & 0.577 & 0.465 & 0.670 & 0.560 \\

    &\metric{SMD + ELMO + PMeans} & 0.631 & 0.472 & 0.631 & 0.499 & 0.663 & 0.498 & 0.726 & 0.568 \\

    &\metric{SMD + BERT + PMeans} & 0.658 & 0.530 & 0.664 & 0.550 & 0.670 & 0.518 & 0.731 & 0.580 \\
    &\metric{SMD + BERT + MNLI + PMeans} & 0.662 & 0.525 & 0.666 & 0.552 & 0.667 & 0.506 & 0.723 & 0.563 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} & 0.669 & 0.549 & 0.665 & 0.588 & 0.698 & 0.520 & 0.740 & 0.647 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.707 & 0.554 & 0.726 & 0.601 & 0.736 & 0.553 & 0.813 & 0.672 \\
    &\metric{Wmd-1 + BERT + PMeans} & 0.729 & 0.595 & 0.755 & 0.660 & 0.742 & 0.581 & 0.825 & 0.690 \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & \textbf{0.736} & \textbf{0.604} & \textbf{0.760} & \textbf{0.672} & \textbf{0.754} & 0.594 & 0.831 & 0.701 \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.734 & 0.601 & 0.752 & 0.663 & 0.753 & 0.586 & 0.825 & 0.694 \\

		\bottomrule
		\end{tabular}
 		\caption{Pearson $r$ and Spearman $\rho$ correlations with summary-level human judgments on TAC 2008 and 2009.
 		}
 		\label{tab:tac0809}
\vspace{-0.08in}
	\end{table*}

\subsection{Text Summarization}
\label{sec:summ_results}

We use two summarization datasets from the Text Analysis Conference
(TAC)\footnote{\url{http://tac.nist.gov}}:
TAC-2008 and TAC-2009, which
contain 48 and 44 \emph{clusters}, respectively.
Each cluster includes 10 news articles (on the same topic),
four reference summaries,
and 57 (in TAC-2008) or 55 (in TAC-2009) system
summaries generated by the participating systems.
Each summary (either reference or system) has fewer than 100 words,
and receives two human judgment scores:
the \emph{Pyramid} score~\cite{nenkova-passonneau:2004:HLTNAACL}
and the \emph{Responsiveness} score.
Pyramid measures how many important semantic content units
in the reference summaries are covered by the system summary,
while Responsiveness measures
how well a summary responds to the overall quality combining both content and linguistic quality.

\paragraph{Results}
Tables \ref{tab:tac0809}:
We observe that lexical metrics like ROUGE correlate above-moderate on TAC 2008 and 2009 datasets. In contrast, these metrics perform poorly on other tasks like Dialogue Generation \cite{novikova-etal-2017-need} and Image Captioning \cite{AndersonFJG16}.
Apparently, strict matches on surface forms seems reasonable for \emph{extractive} summarization datasets.
However, we still
see that our word mover metrics, i.e., WMD-1+BERT+MNLI+PMeans,
perform better than or
come close to
even the
supervised metric $S^{3}_{best}$.

\setlength{\tabcolsep}{8.3pt}
\begin{table*}
	\small
	\centering
	\begin{tabular}{l|l|c c c|c c c}
		\toprule
		&& \multicolumn{3}{c}{BAGEL} & \multicolumn{3}{c}{SFHOTEL} \\
        Setting&Metrics & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} \\
		\midrule
		\noalign{\smallskip}

        \multirow{4}{*}{\metric{Baselines}}
        &\metric{BLEU-1} & 0.225 & 0.141 & 0.113 & 0.107 & 0.175 & 0.069 \\
        &\metric{BLEU-2} & 0.211 & 0.152 & 0.115 & 0.097 & 0.174 & 0.071 \\

        &\metric{METEOR} & 0.251 & 0.127 & 0.116 & 0.111 & 0.148 & 0.082 \\
        &\metric{BERTScore-F1} & 0.267 & 0.210 & \textbf{0.178} & 0.163 & 0.193 & 0.118 \\

    \midrule

    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.024 & 0.074 & 0.078 & 0.022 & 0.025 & 0.011 \\
    &\metric{SMD + ELMO + PMeans} & 0.251 & 0.171 & 0.147 & 0.130 & 0.176 & 0.096 \\

    &\metric{SMD + BERT + PMeans} & 0.290 & 0.163 & 0.121 & 0.192 & 0.223 & 0.134 \\

    &\metric{SMD + BERT + MNLI + PMeans} & 0.280 & 0.149 & 0.120 & 0.205 & 0.239 & 0.147 \\

    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}
    &\metric{Wmd-1 + W2V} & 0.222 & 0.079 & 0.123 & 0.074 & 0.095 & 0.021 \\
    &\metric{Wmd-1 + ELMO + PMeans} & 0.261 & 0.163 & 0.148 & 0.147 & 0.215 & 0.136  \\
    &\metric{Wmd-1 + BERT + PMeans} & \textbf{0.298} & \textbf{0.212} & 0.163 & 0.203 & 0.261 & 0.182  \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.285 & 0.195 & 0.158 & \textbf{0.207} & \textbf{0.270} & \textbf{0.183}  \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.284 & 0.194 & 0.156 & 0.204 & 0.270 & 0.182  \\

		\bottomrule
	\end{tabular}
	\caption{Spearman correlation with utterance-level human judgments for BAGEL and SFHOTEL datasets.
	}
	\label{tab:bag_sfh}
\vspace{-0.1in}
\end{table*}
\setlength{\tabcolsep}{1.4pt}
\subsection{Data-to-text Generation}
\label{sec:gen_results}
We use two task-oriented dialogue datasets: BAGEL \cite{mairesse:2010} and SFHOTEL \cite{wen:2015}, which contains 202 and 398 instances of Meaning Representation (MR). Each MR instance includes multiple references, and roughly two system utterances generated by different neural systems. Each system utterance receives three human judgment scores: \emph{informativeness}, \emph{naturalness} and \emph{quality} score \cite{novikova-etal-2017-need}. Informativeness
measures
how much information a system utterance
provides with respect to an MR. Naturalness measures how likely a system utterance is generated by native speakers. Quality %
measures
how well a system utterance
captures fluency and grammar.
\paragraph{Results}

Tables \ref{tab:bag_sfh}:
Interestingly,
no metric produces an even moderate correlation with human judgments, including our own. We speculate that current contextualizers are poor at representing named entities like hotels and place names as well as numbers appearing in system and reference texts.
However,
best correlation is still achieved by our word mover metrics combining contextualized representations.

\subsection{Image Captioning}
\label{sec:cap_results}
We use a
popular
image captioning dataset: MS-COCO \cite{lin:2014:coco}, which contains 5,000 images. Each image includes roughly five reference captions, and 12 system captions generated by the participating systems from 2015 COCO Captioning Challenge. For the system-level human correlation, each system receives five human judgment scores: M1, M2, M3, M4, M5 \citep{AndersonFJG16}. The M1 and M2 scores
measure overall quality of the captions while M3, M4 and M5 scores
measure correctness, detailedness and saliency of the captions.
Following \citet{cui:2018}, we compare the Pearson correlation with two system-level scores: M1 and M2, since we focus on studying metrics
for the overall quality of the captions, leaving metrics understanding captions in different aspects (correctness, detailedness and saliency) to future work.

\paragraph{Results}
Table \ref{tab:caption}:
Word mover metrics outperform all baselines except for the supervised metric LEIC, which  uses more information by considering \emph{both} images and texts.
\setlength{\tabcolsep}{3.7pt}
\begin{table}[t!]
    \small
    \centering
    \begin{tabular}{l|l | c c}
    \toprule
    Setting & Metric & M1 & M2 \\
    \midrule

    \multirow{5}{*}{\textsc{Baselines}}
    &\metric{LEIC(*)} & \textbf{0.939} & \textbf{0.949} \\
    &\metric{METEOR} & 0.606 & 0.594 \\

    &\metric{SPICE} & 0.759 & 0.750 \\
    &\metric{BERTScore-Recall} & 0.809 & 0.749\\

    \midrule
    \multirow{4}{*}{\metric{Sent-Mover}}
    &\metric{SMD + W2V} & 0.683 & 0.668\\
    &\metric{SMD + ELMO + P} & 0.709 & 0.712\\
    &\metric{SMD + BERT + P} & 0.723 & 0.747\\
    &\metric{SMD + BERT + M + P} & 0.789 & 0.784\\
    \midrule

    \multirow{5}{*}{\metric{Word-Mover}}

    &\metric{Wmd-1 + W2V} & 0.728 & 0.764\\
    &\metric{Wmd-1 + ELMO + P} & 0.753 & 0.775\\
    &\metric{Wmd-1 + BERT + P} & 0.780 & 0.790\\
    &\metric{Wmd-1 + BERT + M + P} & \textbf{0.813} & \textbf{0.810}\\
    &\metric{Wmd-2 + BERT + M + P} & 0.812 & 0.808\\

    \bottomrule
    \end{tabular}
    \caption{Pearson correlation with system-level human judgments on MSCOCO dataset. 'M' and 'P' are short names.}
    \label{tab:caption}
\vspace{-0.05in}
\end{table}

\setlength{\tabcolsep}{5pt}
\begin{table}[h!]
    \footnotesize
    \centering
    \begin{tabular}{l|cccc}
    \toprule

    Metrics & cs-en & de-en & fi-en & lv-en \\

    \midrule

    \metric{RUSE} & 0.624 & 0.644 & 0.750 & 0.697 \\
    \midrule

    \metric{\metric{Hmd-F1} + BERT} & 0.655 & 0.681 & 0.821 & 0.712 \\
    \metric{\metric{Hmd-Recall} + BERT} & 0.651 & 0.658 & 0.788 & 0.681\\
    \metric{\metric{Hmd-Prec} + BERT} & 0.624 & 0.669 & 0.817 & 0.707 \\
    \midrule
    \metric{Wmd-unigram + BERT} & 0.651 & 0.686 & \textbf{0.823} & 0.710 \\
    \metric{Wmd-bigram + BERT} & \textbf{0.665} & \textbf{0.688} & 0.821 & \textbf{0.712} \\

    \hline
    \end{tabular}

    \caption{Comparison on hard and soft alignments.}
    \label{tab:wmt17-ablation}
\vspace{-0.05in}
\end{table}

\subsection{Further Analysis}
\noindent\textbf{Hard and Soft Alignments}\quad
BERTScore is the harmonic mean of BERTScore-Precision and BERTScore-Recall, where both two can be
decomposed
as a combination of ``Hard Mover Distance'' (HMD) and BERT (see Prop.~\ref{prop:main}).

We use the representations in the 9-th BERT layer for fair comparison of
BERTScore and MoverScore
and show results on the machine translation task in Table \ref{tab:wmt17-ablation}.
MoverScore outperforms both asymmetric HMD factors, while if they are combined via harmonic mean, BERTScore is on par with MoverScore. We conjecture that BERT softens hard alignments of BERTScore as contextualized embeddings encode information about the whole sentence into each word vector.
We also observe that \metric{Wmd-bigrams} slightly
outperforms \metric{Wmd-unigrams} on 3 out of 4 language pairs.

\vspace{0.1in}
\noindent\textbf{Distribution of Scores}\quad
In Figure \ref{fig:score_dist}, we take a closer look at sentence-level correlation in MT.
Results reveal that the lexical metric \metric{SentBLEU} can correctly assign lower scores to system translations of low quality, while it struggles in judging system translations of high quality by assigning them lower scores. Our finding agrees with the observations found in \citet{chaganty:2018,novikova-etal-2017-need}: lexical metrics correlate better with human judgments on texts of low quality than high quality.
\citet{peyrard-2019-studying} further show that lexical metrics
cannot be trusted because they strongly disagree on high-scoring system outputs.
Importantly, we observe that our word mover metric combining BERT can clearly distinguish texts of two polar qualities.

\begin{figure}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/score-dist-SentBLEU.pdf}}
\end{minipage}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/score-dist-BERT.pdf}}
\end{minipage}
 \caption{
 Score distribution in German-to-English pair.
 }
 \label{fig:score_dist}
\vspace{-0.1in}
\end{figure}

\begin{figure}
\begin{minipage}{0.49\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/corrplot_de_en.pdf}}
\end{minipage}
\begin{minipage}{0.48\linewidth}
	\centerline{\includegraphics[width=\linewidth]{figures/corrplot_zh_en.pdf}}
\end{minipage}
 \caption{
 Correlation in similar language (de-en) and distant language (zh-en) pair, where bordered area shows correlations between human assessment and metrics, the rest shows inter-correlations across metrics and DA is direct assessment rated by language experts.
 }
 \label{fig:color_example}
\vspace{-0.15in}
\end{figure}

\vspace{0.1in}
\noindent\textbf{Correlation Analysis}\quad
In Figure \ref{fig:color_example}, we observe existing metrics for MT evaluation attaining medium  correlations (0.4-0.5) with human judgments but high inter-correlations between themselves.
In contrast,
our metrics
can attain high correlations (0.6-0.7) with human judgments, performing robust across different language pairs.
We believe that our improvements come from clearly distinguishing translations that fall on two extremes.

\vspace{0.1in}
\noindent\textbf{Impact of Fine-tuning Tasks}\quad
Figure \ref{fig:task-transfer-analysis} compares Pearson correlations with our word mover metrics combining BERT fine-tuned on three different tasks. We observe that fine-tuning on closely related tasks improves correlations, especially fine-tuning on MNLI leads to an impressive improvement by 1.8 points on average.
\begin{figure}[t]
\centering
\includegraphics[width=.6\linewidth]{figures/task-transfer-wmt17.pdf} %
\caption{Correlation is averaged over 7 language pairs.}
 \label{fig:task-transfer-analysis}
\vspace{-0.05in}
\end{figure}

\input{discussion.tex}

\subsection{Proof of Prop.~\ref{prop:main}}

In this section, we prove Prop.~\ref{prop:main} in the paper about viewing BERTScore (precision/recall) as a (non-optimized) Mover Distance.

As a reminder, the WMD formulation is:
\begin{align*}
& \text{WMD}(\boldsymbol{x}^n,\boldsymbol{y}^n) := \min_{{\bm{F}}\in\mathbb{R}^{|\boldsymbol{x}^n|\times |\boldsymbol{y}^n|}}\sum_{i,j} {\bm{C}}_{ij} \cdot {\bm{F}}_{ij}\\
&\text{s.t. } \boldsymbol{1}^{\intercal}{\bm{F}}^{\intercal}\boldsymbol{1}=1,\;\;\boldsymbol{1}^{\intercal}{\bm{F}}\boldsymbol{1}=1.
\end{align*}
where ${\bm{F}}^{\intercal}\boldsymbol{1}=\boldsymbol{f}_x^n$ and ${\bm{F}}\boldsymbol{1}=\boldsymbol{f}_y^n$. Here, $\boldsymbol{f}_x^n$ and $\boldsymbol{f}_y^n$ denote vectors of weights for each $n$-gram of $\boldsymbol{x}^n$ and $\boldsymbol{y}^n$.

BERTScore is defined as:
\begin{align}
R_\bert  &= \frac{\sum_{y_i^1 \in \boldsymbol{y}^1} {\rm idf}(y_i^1)  \max_{x_j^1 \in \boldsymbol{x}^1} E(x_j^1)^\intercal E(y_i^1)}{\sum_{y_i^1\in \boldsymbol{y}^1} {\rm idf}(y_i^1)} \nonumber \\
P_\bert  &= \frac{\sum_{x_j^1 \in \boldsymbol{x}^1} {\rm idf}(x_j^1)  \max_{y_i^1 \in \boldsymbol{y}^1} E(y_i^1)^\intercal E(x_j^1)}{\sum_{x_j^1\in \boldsymbol{x}^1} {\rm idf}(x_j^1)} \nonumber \\
F_\bert &= 2\frac{P_\bert \cdot R_\bert }{ P_\bert + R_\bert }\;\;. \nonumber
\end{align}
Then, $R_\bert$ can be formulated in a ``quasi'' WMD form:

\begin{align*}
& R_\bert(\boldsymbol{x}^1,\boldsymbol{y}^1) := \sum_{i,j} {\bm{C}}_{ij} \cdot {\bm{F}}_{ij}\\
&{\bm{F}}_{ij}=\begin{cases}
      \frac{1}{M} &

      \text{if } x_j=\argmax_{\hat{x}_j^1\in \boldsymbol{x}^1}E(y_i^1)^\intercal E(\hat{x}_j^1)\\
      0 & \text{otherwise}
\end{cases}\\
&{\bm{C}}_{ij}=\begin{cases}
      \frac{M}{Z}{\rm idf}(y_i^1) E(x_j^1)^\intercal E(y_i^1) & \text{if } x_j=\argmax_{\hat{x}_j^1\in \boldsymbol{x}^1}E(y_i^1)^\intercal E(\hat{x}_j^1)\\
      0 & \text{otherwise}
\end{cases}\\
\end{align*}
where $Z=\sum_{y_i^1 \in \boldsymbol{y}^1}{\rm idf}(y_i^1)$ and $M$ is the size of $n$-grams in $\boldsymbol{x}^1$. Similarly, we can have $P_\bert$ in a quasi WMD form (omitted). Then, $F_\bert$ can be formulated as harmonic-mean of two WMD forms of $P_\bert$ and $R_\bert$.

\subsection{Routing}
In this section, we study the aggregation function $\phi$ with a routing scheme, which has achieved good results in other NLP tasks \cite{zhao:2018, zhao-etal-2019-towards}.
Specifically, we introduce a nonparametric clustering with Kernel Density Estimation (KDE) for routing since KDE bridges a family of kernel functions with underlying empirical distributions, which often leads to computational efficiency \cite{zhangfast}, defined as:
\begin{align*}
&\min_{\boldsymbol{v},\gamma}
f(\boldsymbol{z})
=\sum_{i=1}^L \sum_{j=1}^T\gamma_{ij}k(d(\boldsymbol{v}_j - \boldsymbol{z}_{i,j}))\\
& \textit{s.t.}\quad\forall{i,j}:\gamma_{ij}>0,\sum\limits_{j=1}^{L}\gamma_{ij}=1.
\end{align*}
where $d(\cdot)$ is a distance function, $\gamma_{ij}$ denotes the  underlying closeness between the aggregated vector $\boldsymbol{v}_j$ and vector $\boldsymbol{z}_i$ in the $i$-th layer, and $k$ is a kernel function. Some instantiations of $k(\cdot)$ \cite{wand:1994} are:
\begin{align*}
\mathit{Gaussian}: k(x)\triangleq\exp{(-\frac{x}{2})},\;\;
\mathit{Epanechnikov}: k(x)\triangleq\left\{\begin{aligned}1-x && x\in{[0,1)}\\ 0 && x\ge{1}.\end{aligned}\right.
\end{align*}
One typical solution for KDE clustering to minimize $f(\boldsymbol{z})$ is taking Mean Shift \cite{comaniciu:2002}, defined as:
\begin{align*}
	\nabla f(\boldsymbol{z})=\sum_{i,j} \gamma_{ij} k'(d(\boldsymbol{v}_j, \boldsymbol{z}_{i,j}))\frac{\partial{d(\boldsymbol{v}_j,  \boldsymbol{z}_{i,j})}}{\partial{\boldsymbol{v}}} \nonumber
\end{align*}
Firstly, $\boldsymbol{v}_j^{\tau+1}$ can be updated while $\gamma_{ij}^{\tau+1}$ is fixed:
\begin{align*}
	\boldsymbol{v}_j^{\tau+1} = \frac{\sum_{i} \gamma_{ij}^\tau k'(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j})) \boldsymbol{z}_{i,j}}{\sum_{i,j} k'(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j}))}
\end{align*}
Intuitively, $\boldsymbol{v}_j$ can be explained as a final aggregated vector from $L$ contextualized layers.
Then, we adopt SGD to update $\gamma_{ij}^{\tau+1}$:
\begin{align*}
	\gamma_{ij}^{\tau+1} = \gamma_{ij}^\tau + \alpha \cdot k(d(\boldsymbol{v}_j^\tau, \boldsymbol{z}_{i,j}))
\end{align*}
where $\alpha$ is a hyperparameter to control step size. The routing process is summarized in Algorithm \ref{alg:1}.

\begin{algorithm}
\caption{Aggregation by Routing}\label{alg:1}
\begin{algorithmic}[1]
\small
\State \textbf{procedure} ROUTING($\boldsymbol{z}_{ij}$, $\ell$)
\State Initialize $\forall i,j: \gamma_{ij}=0$
\While {true}
  \ForAll{representation $i$ and $j$ in layer $\ell$ and  $\ell+1$}

    $\gamma_{ij} \gets \mathit{softmax}(\gamma_{ij})$
  \EndFor

  \ForAll{representation $j$ in layer $\ell+1$}
    \State $\boldsymbol{v}_j \gets \sum_{i} \gamma_{ij} k'(\boldsymbol{v}_j, \boldsymbol{z}_i) \boldsymbol{z}_i / \sum_{i} k'(\boldsymbol{v}_i, \boldsymbol{z}_i)$
  \EndFor

  \ForAll{representation $i$ and $j$ in layer $\ell$ and  $\ell+1$} $\gamma_{ij}\gets \gamma_{ij} + \alpha \cdot k(\boldsymbol{v}_j, \boldsymbol{z}_i)$

  \EndFor

  \State $\rm{loss} \gets \log(\sum_{i,j}\gamma_{ij}k(\boldsymbol{v}_j, \boldsymbol{z}_i))$
  \If{$|\rm{loss}- \rm{preloss}| < \epsilon$}
      \State \textbf{break}
  \Else
      \State $\rm{preloss} \gets \rm{loss}$
  \EndIf
\EndWhile
\State \textbf{return} $v_j$
\end{algorithmic}
\end{algorithm}

\paragraph{Best Layer and Layer-wise Consolidation}
Table \ref{tab:wmt17-aggregation} compares our word mover based metric combining BERT representations on different layers with stronger BERT representations consolidated from these layers (using $p$-means and routing). We often see that which layer has best performance is task-dependent, and our word mover based metrics (WMD) with $p$-means or routing schema come close to the oracle performance obtained from the best layers.

\setlength{\tabcolsep}{8.8pt}
\begin{table*}[h!]
    \small
    \centering
    \begin{tabular}{l ccccccc}
    \toprule
    & \multicolumn{7}{c}{\textbf{Direct Assessment}}\\
    Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en \\

    \midrule

    \metric{Wmd-1 + BERT + Layer 8} & .6361 & .6755 & .8134 & .7033 & .7273 & .7233 & .7175 \\
    \metric{Wmd-1 + BERT + Layer 9} & .6510 & .6865 & .8240 & .7107 & .7291 & \textbf{.7357} & \textbf{.7195} \\
    \metric{Wmd-1 + BERT + Layer 10} & .6605 & \textbf{.6948} & \textbf{.8231} & .7158 & \textbf{.7363} & .7317 & .7168 \\
    \metric{Wmd-1 + BERT + Layer 11} & \textbf{.6695} & .6845 & .8192 & .7048 & .7315 & .7276 & .7058 \\
    \metric{Wmd-1 + BERT + Layer 12} & .6677 & .6825 & .8194 & \textbf{.7188} & .7326 & .7291 & .7064 \\
    \midrule

    \metric{Wmd-1 + BERT + Routing} & .6618 & .6897 & .8225 & .7122 & .7334 & .7301 & .7182 \\
    \metric{Wmd-1 + BERT + PMeans} & .6623 & .6873 & .8234 & .7139 & .7350 & .7339 & .7192 \\

	\hline
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations.}
    \label{tab:wmt17-aggregation}
\end{table*}

\paragraph{Experiments}
Table ~\ref{tab:wmt17-all}, \ref{tab:tac-all} and \ref{tab:dialogue-all} show correlations between metrics (all baseline metrics and word mover based metrics) and human judgments on machine translation, text summarization and dialogue response generation, respectively. We find that word mover based metrics combining BERT fine-tuned on MNLI
have highest correlations with humans,
outperforming
all of the unsupervised metrics and even
supervised metrics like $\metric{RUSE}$ and $\metric{$S^{3}_{full}$}$.
Routing and $p$-means
perform roughly equally well.

\setlength{\tabcolsep}{5pt}
\begin{table}[h!]
    \footnotesize
    \centering
    \begin{tabular}{c|l|ccccccc c}
    \toprule
    & & \multicolumn{8}{c}{\textbf{Direct Assessment}}\\
    Setting & Metrics & cs-en & de-en & fi-en & lv-en & ru-en & tr-en & zh-en & Average \\

    \midrule
    \multirow{5}{*}{\metric{Baselines}}
    & \metric{BLEND} & 0.594 & 0.571 & 0.733 & 0.594 & 0.622 & 0.671 & 0.661 & 0.635 \\
    & \metric{RUSE} & 0.624 & 0.644 & 0.750 & 0.697 & 0.673 & 0.716 & 0.691 & 0.685 \\

    & \metric{SentBLEU} & 0.435 & 0.432 & 0.571 & 0.393 & 0.484 & 0.538 & 0.512 & 0.481 \\
    & \metric{chrF++} & 0.523 & 0.534 & 0.678 & 0.520 & 0.588 & 0.614 & 0.593 & 0.579 \\
    & \metric{METEOR++} & 0.552 & 0.538 & 0.720 & 0.563 & 0.627 & 0.626 & 0.646 & 0.610 \\
    & \metric{BERTScore-F1} & 0.670 & 0.686 & 0.820 & 0.710 & 0.729 & 0.714 & 0.704 & 0.719 \\
    \midrule

    \multirow{7}{*}{\metric{Word-Mover}}

    & \metric{Wmd-1 + W2V} &0.392 & 0.463 & 0.558 & 0.463 & 0.456 & 0.485 & 0.481 & 0.471 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.658 & 0.689 & 0.823 & 0.712 & 0.733 & 0.730 & 0.718 & 0.723 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.665 & 0.705 & 0.834 & 0.744 & 0.735 & 0.752 & 0.736 & 0.739 \\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.676 & 0.706 & 0.831 & 0.743 & 0.734 & 0.755 & 0.732 & 0.740 \\

    &\metric{Wmd-1 + BERT + PMeans} & 0.662 & 0.687 & 0.823 & 0.714 & 0.735 & 0.734 & 0.719 & 0.725\\

    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.670 & 0.708 & \textbf{0.835} & \textbf{0.746} & \textbf{0.738} & 0.762 & \textbf{0.744} & \textbf{0.743}\\

    &\metric{Wmd-2 + BERT + MNLI + PMeans} & \textbf{0.679} & \textbf{0.710} & 0.832 & 0.745 & 0.736 & \textbf{0.763} & 0.740 & \textbf{0.743}\\

	\hline
    \end{tabular}

    \caption{Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations.}
    \label{tab:wmt17-all}
\end{table}

\setlength{\tabcolsep}{4.5pt}
\begin{table}
		\small
		\centering
  		\begin{tabular}{c|l |cc|cc|cc|cc}
  		\toprule
 		&&\multicolumn{4}{c|}{TAC-2008}	&	\multicolumn{4}{c}{TAC-2009}\\
 		&&\multicolumn{2}{c|}{\textbf{Responsiveness}}&\multicolumn{2}{c|}{\textbf{Pyramid}} &\multicolumn{2}{c|}{\textbf{Responsiveness}} & \multicolumn{2}{c}{\textbf{Pyramid}}\\
 		Setting & Metrics	&	$r$ & $\rho$& $r$   & $\rho$ & $r$ & $\rho$ & $r$   &$\rho$\\
 		\midrule

        \multirow{13}{*}{\metric{Baselines}}
        &\metric{$S^{3}_{full}$} & 0.696 & 0.558 & 0.753 & 0.652 & 0.731 & 0.552 & 0.838 & 0.724 \\
		&\metric{$S^{3}_{best}$} & 0.715 & 0.595 & 0.754 & 0.652 & 0.738 & \textbf{0.595} & \textbf{0.842} & \textbf{0.731} \\

		&\metric{TF$*$IDF-1}	& 0.176 & 0.224 & 0.183 & 0.237 & 0.187 & 0.222 & 0.242 & 0.284\\
		&\metric{TF$*$IDF-2}	& 0.047 & 0.154 & 0.049 & 0.182 & 0.047 & 0.167 & 0.097 & 0.233\\
		&\metric{ROUGE-1} & 0.703 & 0.578 & 0.747 & 0.632 & 0.704 & 0.565 & 0.808 & 0.692 \\
		&\metric{ROUGE-2} & 0.695 & 0.572 & 0.718 & 0.635 & 0.727 & 0.583 & 0.803 & 0.694 \\
		&\metric{ROUGE-1-WE}	& 0.571 & 0.450 & 0.579 & 0.458 & 0.586 & 0.437 & 0.653 & 0.516 \\
		&\metric{ROUGE-2-WE}	& 0.566 & 0.397 & 0.556 & 0.388 & 0.607 & 0.413 & 0.671 & 0.481 \\
		&\metric{ROUGE-L} & 0.681 & 0.520 & 0.702 & 0.568 & 0.730 & 0.563 & 0.779 & 0.652 \\
		&\metric{Frame-1} & 0.658 & 0.508 & 0.686 & 0.529 & 0.678 & 0.527 & 0.762 & 0.628 \\
		&\metric{Frame-2} & 0.676 & 0.519 & 0.691 & 0.556 & 0.715 & 0.555 & 0.781 & 0.648 \\
        & \metric{BERTScore-F1} & 0.724 & 0.594 & 0.750 & 0.649 & 0.739 & 0.580 & 0.823 & 0.703 \\
		\midrule

    \multirow{7}{*}{\metric{Word-Mover}}

    & \metric{Wmd-1 + W2V} & 0.669 & 0.559 & 0.665 & 0.611 & 0.698 & 0.520 & 0.740 & 0.647 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.729 & 0.601 & 0.763 & 0.675 & 0.740 & 0.580 & 0.831 & 0.700 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.734 & \textbf{0.609} & \textbf{0.768} & \textbf{0.686} & 0.747 & 0.589 & 0.837 & 0.711 \\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.731 & 0.593 & 0.755 & 0.666 & 0.753 & 0.583 & 0.827 & 0.698 \\
    &\metric{Wmd-1 + BERT + PMeans} & 0.729 & 0.595 & 0.755 & 0.660 & 0.742 & 0.581 & 0.825 & 0.690 \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & \textbf{0.736} & 0.604 & 0.760 & 0.672 & \textbf{0.754} & 0.594 & 0.831 & 0.701 \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.734 & 0.601 & 0.752 & 0.663 & 0.753 & 0.586 & 0.825 & 0.694 \\

		\hline
		\end{tabular}
 		\caption{Correlation of automatic metrics with summary-level human judgments for TAC-2008 and TAC-2009.}
 		\label{tab:tac-all}
	\end{table}
\vspace{-0.1in}

\setlength{\tabcolsep}{8pt}
\begin{table}
    \small
	\centering

	\begin{tabular}{c | l|c c c|c c c}
		\hline\noalign{\smallskip}
		&& \multicolumn{3}{c}{BAGEL} & \multicolumn{3}{c}{SFHOTEL} \\
        Setting & Metrics & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} & \textbf{Inf} & \textbf{Nat} & \textbf{Qual} \\
		\hline\noalign{\smallskip}

        \multirow{8}{*}{\metric{Baselines}}
        &\metric{BLEU-1} & 0.225 & 0.141 & 0.113 & 0.107 & 0.175 & 0.069 \\
        &\metric{BLEU-2} & 0.211 & 0.152 & 0.115 & 0.097 & 0.174 & 0.071 \\
        &\metric{BLEU-3} & 0.191 & 0.150 & 0.109 & 0.089 & 0.161 & 0.070 \\
        &\metric{BLEU-4} & 0.175 & 0.141 & 0.101 & 0.084 & 0.104 & 0.056 \\
        &\metric{ROUGE-L} & 0.202 & 0.134 & 0.111 & 0.092 & 0.147 & 0.062 \\
        &\metric{NIST} & 0.207 & 0.089 & 0.056 & 0.072 & 0.125 & 0.061 \\
        &\metric{CIDEr} & 0.205 & 0.162 & 0.119 & 0.095 & 0.155 & 0.052 \\
        &\metric{METEOR} & 0.251 & 0.127 & 0.116 & 0.111 & 0.148 & 0.082 \\
        & \metric{BERTScore-F1} & 0.267 & 0.210 & \textbf{0.178} & 0.163 & 0.193 & 0.118 \\

    \midrule
    \multirow{7}{*}{\metric{Word-Mover}}
    & \metric{Wmd-1 + W2V} & 0.222 & 0.079 & 0.123 & 0.074 & 0.095 & 0.021 \\

    & \metric{Wmd-1 + BERT + Routing} & 0.294 & 0.209 & 0.156 & 0.208 & 0.256 & 0.178 \\

    & \metric{Wmd-1 + BERT + MNLI + Routing} & 0.278 & 0.180 & 0.144 & \textbf{0.211} & 0.252 & 0.175\\
    & \metric{Wmd-2 + BERT + MNLI + Routing} & 0.279 & 0.182 & 0.147 & 0.204 & 0.252 & 0.172 \\
    &\metric{Wmd-1 + BERT + PMeans} & \textbf{0.298} & \textbf{0.212} & 0.163 & 0.203 & 0.261 & 0.182  \\
    &\metric{Wmd-1 + BERT + MNLI + PMeans} & 0.285 & 0.195 & 0.158 & 0.207 & \textbf{0.270} & \textbf{0.183}  \\
    &\metric{Wmd-2 + BERT + MNLI + PMeans} & 0.284 & 0.194 & 0.156 & 0.204 & \textbf{0.270} & 0.182  \\

		\hline
	\end{tabular}
	\caption{Spearman correlation with utterance-level human judgments for BAGEL and SFHOTEL datasets.}
	\label{tab:dialogue-all}
\end{table}
\setlength{\tabcolsep}{1.4pt}

redpajama_set_name: RedPajamaArXiv


text:
Effect of substitutes in contingent valuation for a private market good
Ji Yong Lee, John A. Fox, Rodolfo M. Nayga
Economic theory predicts a decrease in valuation as the availability of substitutes increases. This paper describes a contingent valuation (CV) survey that investigates the effect of substitutes on valuation of private market goods. Using an approach that compares willingness to pay (WTP) values elicited from a CV question that accounts for substitutes with WTP values elicited from a similar question without substitutes, we find that allowing for substitutes can moderate WTP values. For the item valued in this study, a hamburger sandwich, allowing for substitutes was associated with a reduction of from 10% to 16% in stated values.
Applied Economics Letters
Contingent valuation
field substitutes
market good
Dive into the research topics of 'Effect of substitutes in contingent valuation for a private market good'. Together they form a unique fingerprint.
Contingent Valuation Business & Economics 100%
Substitute Business & Economics 72%
Willingness-to-pay Business & Economics 70%
Economic Theory Business & Economics 12%
Goods Business & Economics 9%
Lee, J. Y., Fox, J. A., & Nayga, R. M. (2019). Effect of substitutes in contingent valuation for a private market good. Applied Economics Letters, 26(14), 1153-1156. https://doi.org/10.1080/13504851.2018.1540837
Effect of substitutes in contingent valuation for a private market good. / Lee, Ji Yong; Fox, John A.; Nayga, Rodolfo M.
In: Applied Economics Letters, Vol. 26, No. 14, 16.08.2019, p. 1153-1156.
Lee, JY, Fox, JA & Nayga, RM 2019, 'Effect of substitutes in contingent valuation for a private market good', Applied Economics Letters, vol. 26, no. 14, pp. 1153-1156. https://doi.org/10.1080/13504851.2018.1540837
Lee JY, Fox JA, Nayga RM. Effect of substitutes in contingent valuation for a private market good. Applied Economics Letters. 2019 Aug 16;26(14):1153-1156. doi: 10.1080/13504851.2018.1540837
Lee, Ji Yong ; Fox, John A. ; Nayga, Rodolfo M. / Effect of substitutes in contingent valuation for a private market good. In: Applied Economics Letters. 2019 ; Vol. 26, No. 14. pp. 1153-1156.
@article{cb5a0653fdbb44b8ad2720f3d2bdd9a4,
title = "Effect of substitutes in contingent valuation for a private market good",
abstract = "Economic theory predicts a decrease in valuation as the availability of substitutes increases. This paper describes a contingent valuation (CV) survey that investigates the effect of substitutes on valuation of private market goods. Using an approach that compares willingness to pay (WTP) values elicited from a CV question that accounts for substitutes with WTP values elicited from a similar question without substitutes, we find that allowing for substitutes can moderate WTP values. For the item valued in this study, a hamburger sandwich, allowing for substitutes was associated with a reduction of from 10% to 16% in stated values.",
keywords = "Contingent valuation, field substitutes, market good",
author = "Lee, {Ji Yong} and Fox, {John A.} and Nayga, {Rodolfo M.}",
note = "Funding Information: This work was supported by the U.S. Department of Agriculture-National Institute of Food and Agriculture (USDA-NIFA) Coordinated Agricultural Project (CAP) grant, titled Shiga-toxigenic Escherichia coli (STEC) in the Beef Chain: Assessing and Mitigating the Risk by Translational Science, Education and Outreach [Award # 2012-68003-30155]. Publisher Copyright: {\textcopyright} 2018, {\textcopyright} 2018 Informa UK Limited, trading as Taylor & Francis Group.",
journal = "Applied Economics Letters",
T1 - Effect of substitutes in contingent valuation for a private market good
AU - Lee, Ji Yong
AU - Fox, John A.
AU - Nayga, Rodolfo M.
N1 - Funding Information: This work was supported by the U.S. Department of Agriculture-National Institute of Food and Agriculture (USDA-NIFA) Coordinated Agricultural Project (CAP) grant, titled Shiga-toxigenic Escherichia coli (STEC) in the Beef Chain: Assessing and Mitigating the Risk by Translational Science, Education and Outreach [Award # 2012-68003-30155]. Publisher Copyright: © 2018, © 2018 Informa UK Limited, trading as Taylor & Francis Group.
N2 - Economic theory predicts a decrease in valuation as the availability of substitutes increases. This paper describes a contingent valuation (CV) survey that investigates the effect of substitutes on valuation of private market goods. Using an approach that compares willingness to pay (WTP) values elicited from a CV question that accounts for substitutes with WTP values elicited from a similar question without substitutes, we find that allowing for substitutes can moderate WTP values. For the item valued in this study, a hamburger sandwich, allowing for substitutes was associated with a reduction of from 10% to 16% in stated values.
AB - Economic theory predicts a decrease in valuation as the availability of substitutes increases. This paper describes a contingent valuation (CV) survey that investigates the effect of substitutes on valuation of private market goods. Using an approach that compares willingness to pay (WTP) values elicited from a CV question that accounts for substitutes with WTP values elicited from a similar question without substitutes, we find that allowing for substitutes can moderate WTP values. For the item valued in this study, a hamburger sandwich, allowing for substitutes was associated with a reduction of from 10% to 16% in stated values.
KW - Contingent valuation
KW - field substitutes
KW - market good
JO - Applied Economics Letters
JF - Applied Economics Letters

redpajama_set_name: RedPajamaCommonCrawl