\section{Introduction}

Pre-trained Language Models (PLMs) have achieved promising results in many Chinese Natural Language Understanding (NLU) tasks \cite{bert-wwm,kbert,ernie2}.
These models take a sequence of fine-grained units --- Chinese characters --- as the input, following the English PLMs' practice \cite[BERT]{bert}.

However, the meanings of many Chinese words cannot be fully understood through direct compositions of their characters' meanings.
For example, 老板/\textit{boss} does not mean 老/\textit{elder} 板/\textit{board}.\footnote{For clarity, we use 中文/\textit{English translation} to represent an example in Chinese with its \textit{translation} followed by.}
The importance of word-level inputs in Chinese has been addressed in different tasks, including relation classification
\cite{li2019chinese}, short text matching \cite{lai2019lattice,chen-etal-2020-neural,Lyu2021LETLK}, trigger detection \cite{lin2018nugget}, and named entity recognition \cite{latticeLSTMner,gui2019lexicon,flat}.
The coarse-grained inputs benefit these tasks by introducing word-level semantics with multi-granularity representations,
which is potentially complementary in character-level Chinese PLMs.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.97\columnwidth]{lattice.pdf}
\caption{ An illustration of the word lattice for sentence 研究生活很充实/\textit{Research Life is very fulfilling}.
\label{fig:lattice}}
\end{figure}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.97\textwidth]{lattice-bert-2.pdf}
\caption{An illustration of our pre-training framework, Lattice-BERT.
\label{fig:lattice-bert}
}
\end{figure*}

In this work, we discuss how to pre-train a Chinese PLM over a \textit{word lattice} structure to exploit multi-granularity inputs.
We argue that by incorporating the coarse-grained units into PLM,
models could learn to utilize the multi-granularity information for downstream tasks.
Specifically, we organize characters and words in sentences as word lattices (see Figure~\ref{fig:lattice}), which enable the models to explore the words from all possible word segmentation results.

However, it is not straightforward to learn a BERT-like PLM over the word lattices.
The major challenges are two-folded.
Firstly, BERT's original input is a sequence of characters ordered by their positions, making it difficult to consume the word lattices and preserve the positional relationship between multi-granularity units.
Secondly, the conventional masked language modeling (MLM) task may make the word-lattice based PLMs learn unexpected tricks. The reason is that such a word lattice naturally introduces redundancy, that is, one character can be contained in multiple text units.
In MLM, models may refer to the other text units overlapping with the randomly masked one instead of the real context, which brings information leakages.

To address these challenges, we propose a Lattice-based Bidirectional Encoder Representation from Transformers (Lattice-BERT).
Specifically, we design a \textit{lattice position attention} (LPA) to help the transformers directly exploit positional relationship and distances between text units in lattices.
Moreover, we propose a \textit{masked segment prediction} (MSP) task to avoid the potential leakage between overlapping text units in language modeling.
With LPA and MSP, the Lattice-BERT could
harness the multi-granularity structures in lattices, thus,
directly utilize the lattice structures to aggregate the coarse-grained word information to benefit various downstream tasks.

We evaluate our model on 11 Chinese NLU tasks in various paradigms, including the CLUE benchmarks \cite{CLUEbenchmark} as well as two sequence labeling tasks.
Compared with the baseline that only takes characters as inputs, Lattice-BERT bring an average increase of 1.5\% and 2.0\% under the settings of 12 and 6 layers, respectively.
The 12-layer Lattice-BERT model beats all other \textit{base}-size models on CLUE benchmarks.\footnote{\url{https://www.cluebenchmarks.com/rank.html}, until Oct. 31st, 2020.}
Morever, we show that Lattice-BERT can harness the multi-granularity inputs and utilize word-level semantics to outperform vanilla fine-grained PLMs.

Our contributions can be summarized as
1) We propose Lattice-BERT to leverage multi-granularity representations from word lattices in Chinese PLMs.
2) We design \textit{lattice position attention} and \textit{masked segment prediction} to facilitate Chinese PLMs to exploit the lattice structures.
3) Lattice-BERT brings remarkable improvements on 11 Chinese tasks and achieves new state of the arts among \textit{base}-size models at the CLUE benchmarks.

\section{Lattice-BERT}

This section, we detail the implementation of Lattice-BERT, and its overall framework is presented in Figure~\ref{fig:lattice-bert}.

\subsection{Preliminary: BERT}

BERT \cite[Bidirectional Encoder Representations from Transformers]{bert} is a pre-trained language model comprising a stack of multi-head self-attention layers and fully connected layers.
For each head in the $l_{th}$ multi-head self-attention layer, the output matrix $\mathbf{H}^{\mathrm{out}, l} =\left\{\mathbf{h}_1^{\mathrm{out}, l}, \mathbf{h}_2^{\mathrm{out}, l} , \dots, \mathbf{h}_{n}^{\mathrm{out}, l} \right\} \in \mathbb{R}^{n \times d_k} $ satisfies:
$$
\mathbf{h}_i^{\mathrm{out}, l} =
\sum_{j=1}^{n}{\left(\frac{\exp{\alpha_{ij}^l}}{\sum_{j'}{\exp{\alpha_{ij'}^l}}} \mathbf{h}_j^{\mathrm{in},l} \mathbf{W}^{v, l} \right)}
$$
\begin{equation}
\alpha_{ij}^l=\frac{1}{\sqrt{2d_k}}\left(\mathbf{h}_i^{\mathrm{in},l}\mathbf{W}^{q, l}\right){\left(\mathbf{h}_j^{\mathrm{in},l}\mathbf{W}^{k, l}\right)}^T \label{eq1}
\end{equation}
where $\mathbf{H}^{\mathrm{in}, l} =\left\{\mathbf{h}_1^{\mathrm{in}, l}, \mathbf{h}_2^{\mathrm{in}, l} , \dots, \mathbf{h}_n^{\mathrm{in}, l} \right\} \in \mathbb{R}^{n \times d_h}$ is the input matrix, and $\mathbf{W}^{q, l}, \mathbf{W}^{k, l}, \mathbf{W}^{v, l} \in \mathbb{R}^{d_h \times d_k}$ are learnable parameters. $n$ and $d_h$ are sequence length and hidden size, and the attention size $d_k$ = $d_h / n_h$, where $n_h$ is the number of attention heads.

To capture the sequential features in languages, previous PLMs adopt position embedding in either input representations \cite{bert,albert} or attention weights \cite{xlnet,wei2019nezha,ke2020rethinking}.
For the input-level position embedding, the inputs of the first layer are $\mathbf{\widetilde{h}}_i^{\mathrm{in},0} = \mathbf{{h}}_i^{\mathrm{in},0} + \mathbf{P}_i$, where $\mathbf{P}_i$ is the embedding of the $i_{th}$ position.
The other works incorporate position information in attention weights, i.e., $\widetilde{\alpha}_{ij}^l={\alpha}_{ij}^l + f\left(i, j\right)$, where $f$ is a function of the position pair $\left(i, j\right)$.

The BERT model is pre-trained on an unlabeled corpus with reconstruction losses, i.e., Masked Language Modeling (MLM) and Next Sentence Prediction (NSP), and then fine-tuned on downstream tasks to solve specific NLU tasks.
Readers could refer to \citet{bert} for details.

\subsection{Multi-granularity Inputs: Word Lattices}
We adopt a word lattice to consume all possible segmentation results of a sentence in one PLM.
Each segmentation can be a mixture of characters and words.
As shown in Figure~\ref{fig:lattice}, a word lattice is a directed acyclic graph, where the nodes are positions in the original sentences, and each directed edge represents a character or a plausible word.
Word lattices incorporate all words and characters so that models could explicitly exploit the inputs of both granularities, despite some of the words are redundant.
In the rest of this work, we use \textit{\textbf{lattice tokens}} to refer to text units, including the characters and words, contained in lattice graphs.

As shown in Figure~\ref{fig:lattice-bert}, we list the lattice tokens in a line and consume these tokens to transformers straightforwardly.
However, the challenges of learning PLMs as BERT over the lattice-like inputs include:
1) encoding the lattice tokens while preserving lattice structures;
2) avoiding potential leakage brought by redundant information.

\subsection{Interaction: Lattice Position Attention}

Since the original BERT is designed for sequence modeling, it is not straightforward for BERT to consume a lattice graph.
The word lattices encode not only the character sequences but also nested and overlapping words from different segmentations.
To accurately incorporate positional information from lattice graphs into the interactions between tokens, we extend the attention-level position embedding and propose \textit{lattice position attention}.

The \textit{lattice position attention} aggregates the attention score of token representations, $\alpha_{ij}$ in Eq.~\ref{eq1}, with three position related attention terms,
encoding the absolute positions, the distance, and the positional relationship, which can be formulated as:
\begin{equation}
\widetilde{\alpha}_{ij} = {\alpha}_{ij} + \mathrm{att}_{ij} + b_{ij} + r_{ij}
\label{eq2}
\end{equation}

The $\mathrm{att}_{ij}$ in Eq.~\ref{eq2} is the attention weight between the absolute positions:
$$
\mathrm{att}_{ij} = \frac{1}{\sqrt{2 d_k}}
\left(\left[\mathbf{P}^S_{s_i};\mathbf{P}^E_{e_i}\right] \mathbf{W}^{q}\right)
\left(\left[\mathbf{P}^S_{s_j};\mathbf{P}^E_{e_j}\right] \mathbf{W}^{k}\right)^T
$$
$\left[ \cdot ; \cdot  \right]$ means the concatenation of vectors. $\mathbf{W}^{q}, \mathbf{W}^{k} \in \mathbb{R}^{2d_e \times d_k}$ are learnable parameters, $d_e$ and $d_k$ are embedding size and attention size.
$s_i, e_i$ are positions of start and end characters of the $i_{th}$ token.
Taking the word  研究/\textit{research} in Figure~\ref{fig:lattice} as an example, it starts at the first character and ends at the second one, thus, its $s_i$ and $e_i$ are 1 and 2, respectively.
$\mathbf{P}^S$ and $\mathbf{P}^E$ are learnable position embedding matrices.
$\mathbf{P}^S_t, \mathbf{P}^E_t \in \mathbb{R}^{d_e} $ is the $t_{th}$ embedding vector of $\mathbf{P}^S$ or $\mathbf{P}^E$.
The $\mathrm{att}_{ij}$ exploit the prior of attention weight between the start and end positions of the token pairs.

The $b_{ij}$ in Eq.~\ref{eq2} is the attention term for the distance between the $i_{th}$ and $j_{th}$ tokens, which consists of four scaling terms considering the combinations of the start and end positions:
$$
b_{ij} =
{b}^{ss}_{{s_j}-{s_i}} +
{b}^{se}_{{s_j}-{e_i}} +
{b}^{es}_{{e_j}-{s_i}} +
{b}^{ee}_{{e_j}-{e_i}}
$$
$b^{ss}_t$ reflects the attention weight brought by the relative distance $t$ between the start positions of two tokens.
The other terms, i.e., $b^{se}_t$, $b^{es}_t$, and $b^{ee}_t$, have similar meanings.
In practice, the distance \textit{t} is clipped into $\left[-128, 128\right]$.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.48\textwidth]{vis.pdf}
\caption{An illustration of the positional relations. Each rectangle represents a lattice token, corresponding to a character span in the original sentence. \textit{T.1} $\sim$ \textit{T.7} are the target tokens with the seven different position relation to the \textit{Source} token.
\label{fig:pos-rel}
}
\end{figure}

$r_{ij}$ in Eq.~\ref{eq2} is a scaling term represents the positional relation between the $i_{th}$ and $j_{th}$ tokens.
We consider seven relations, including (1) self, (2) left and detached, (3) left and overlapped, (4) containing, (5) contained by, (6) right and overlapped, (7) right and detached.
Figure~\ref{fig:pos-rel} shows an illustration of these 7 relations. Formally,
for the $i_{th}$ and $j_{th}$ tokens, they are overlapped means $s_i \leq s_j < e_i \leq e_j$ or $s_j \leq s_i < e_j \leq e_i$, and if $e_i < s_j$ or $e_j < s_i$, they are detached.
If $s_i \leq s_j \leq e_j \leq e_i$ and $i \neq j$, the $i_{th}$ token contains the $j_{th}$ token and the $j_{th}$ token is contained by the $i_{th}$ token.
Intuitively, only two detached tokens can be concurrent in one Chinese word segmentation result. Moreover, the containing relation reflects a sort of lexical hierarchy in the lattices.
We think $r_{ij}$ can explicitly model the positional relations between tokens in lattice graphs.

We argue that the attention scores for distances and token relations capture different aspects of the multi-granularity structures in lattice graphs, thus, meeting the needs of various downstream tasks, such as distance for coreference resolution and positional relation for named entity recognition.
With the information of absolute positions, distances, and positional relations, PLMs could accurately exploit the lattice structures in attention layers.

The \textit{lattice position attention} weights are shared over all layers.
$b_{ij}$, $r_{ij}$, $\mathbf{W}^q$, and $\mathbf{W}^k$ are diverse in different attention heads to capture diverse attention patterns.
We follow \citet{ke2020rethinking} to reset the positional attention scores related to \textttf{[CLS]} tokens, which is the special token as prefix of the input sequences to capture the overall semantics.

\subsection{Pre-training Tasks: Masked Segment Prediction}

Vanilla BERT is trained to predict the randomly masked tokens in the sentences, i.e., the masked language modeling (MLM).
For the case of consuming multi-granularity inputs, the input tokens are redundant
which means a character can occur in its character forms and multiple words it belongs to.
Directly adopting the randomly masking strategy
may simplify the prediction problem in our case because the masked token can be easily guessed
via peeking the unmasked tokens overlapping with the masked one.
Taking the word 研究/\textit{research} in Figure~\ref{fig:lattice-bert} as an example,
supposing the masked input is \texttt{[M]}/究/研究,
the model will consult 研究 rather than the context to predict the masked token, 研.

We investigate this problem and find that the tokens within a minimal \textit{segment} of the lattice provide strong clue for the prediction of other tokens.
A \textit{\textbf{segment}} is a connected subgraph of a lattice where no token exists outside the subgraph that overlaps with any token inside the subgraph.
To identify these minimal segments, we enumerate the character-level tokens in sentence order, checking if all the word-level tokens which contain this character end at this character.
If so, all the tokens containing previous and current characters are considered as a segment, and the next segment starts from the next character, see the example in Figure~\ref{fig:lattice-bert}.
After the segment detection, we propose a \textit{masked segment prediction} (MSP) task as a replacement of the MLM in the original BERT.
In MSP, we mask out all the tokens in a segment and predict all these tokens (see Figure~\ref{fig:lattice-bert}) to avoid the potential leakage.

In addition to MSP, we also pre-train our models with the sentence order prediction (SOP) task in \citet{albert}, where the model predicts whether two consecutive sentences are swapped in inputs.

\subsection{Downstream tasks with Lattice-BERT}

We explore four kinds of downstream tasks, i.e., sentence/sentence-pair classification, multiple choices, sequence labeling, and span selection machine reading comprehension (MRC).
For the sentence/sentence-pair classification, both vanilla and Lattice-BERT classify input instances base on logistic regressions over the representation of \textttf{[CLS]} tokens in the last layer.
The circumstances are similar in multiple choice tasks, where softmax regressions are conducted over the representations of \textttf{[CLS]} tokens to choose the best options.
However, for the span selection MRC, and the sequence labeling tasks like named entity recognition (NER), models need to perform token-wise classification.
Vanilla BERT predicts labels for the input characters, but lattice-BERT has additional words.
In Lattice-BERT, we extract the character chains (word pieces for numbers and English words) from lattices for training and prediction for a fair comparison with vanilla BERT.
Pilot studies show that this strategy performs comparably with the more complex strategies, which supervise the labels over words and obtain a character's label via ensembles of all tokens containing that character.

\subsection{Implementation}

\paragraph{Lattice Construction.}
We construct the word lattices based on a vocabulary consisting of 102K high-frequency open domain words.
All the sub-strings of the input sequence that appear in the vocabulary are considered lattices tokens of the input.
With Aho-Corasick automaton \cite{aho1975efficient}, this construction procedure can complete in linear time to the size of the corpus and the vocabulary.\footnote{Formally, the time complexity is $O((N+M) \times L)$.
$N$ is the corpus size, $M$ is the vocabulary size, and $L$ is the average length of the words in characters, which is a small constant.}
To deal with English words and numbers where the substrings are meaningless, we use the character sequences for those out-of-vocabulary non-Chinese inputs and remain the in-vocabulary words and word pieces.

We construct word lattices using all possible words according to a vocabulary instead of more sophisticated lattice construction strategies.
Previous research efforts~\cite{lai2019lattice,chen-etal-2020-neural,bertmwa} on lattice construction suggests that using all possible words usually yields better performance.
We think an overly-designed lattice construction method may bias our model on certain types of text, and would probably harm the generalization.
So, in our case, we let the model learn by itself to filter the noise introduced by using \textit{all possible words} during pre-training on a large-scale corpus.

\paragraph{Pre-training Details.}
To compare with previous pre-training works, we implement the \textit{base}-size models, which contains 12 layers, 768-dimensional of hidden size, and 12 attention heads.
To demonstrate how lattice gains in shallower architectures and provide lightweight baselines, we also conduct the \textit{lite}-size models with 6 layers, 8 attention heads, and the hidden size of 512.

To avoid the large vocabulary introducing too many parameters in embedding matrix, we adopt the embedding decomposition trick following \citet[ALBERT]{albert}.
Consequently, the parameters of Lattice-BERT is 100M in base-size, only 11\% more than its character-level counterpart (90M), and smaller than the RoBERTa-base \cite{roberta} (102M) and AMBERT \cite{zhang2020ambert} (176M).
The modeling of positional relation and distances in \textit{lattice position attention} introduces only 12K parameters.

A collection of Chinese text, including Chinese Wikipedia, Zhihu, and web news, is used in our BERT models' pre-training stage.
The total number of characters in our unlabeled data is 18.3G.
We follow \citet{roberta} and train the PLMs with a large batch size of 8K instances for 100K steps.
The hyper-parameters and details are given in Appendix~\ref{app:imp}.

\section{Experiments}

We present the details of the Lattice-BERT fine-tuning results on 11 Chinese NLU tasks.
Answering the following questions:
(1) Whether the Lattice-BERT performs better than mono-granularity PLMs and other multi-granularity PLMs?
(2) How the proposed \textit{lattice position attention} and \textit{masked segment prediction} contribute to the downstream tasks?
(3) How Lattice-BERT outperforms the original character-level PLMs?

\subsection{Tasks}
We test our models on 11 Chinese NLU tasks, including the text classification and Machine Reading Comprehension (MRC) tasks in the Chinese Language Understanding Evaluation benchmark \cite[CLUE]{CLUEbenchmark}, and two additional tasks to probe the effectiveness in sequence labeling.

\textbf{CLUE text classification}:
natural language inference \textbf{CMNLI},
long text classification IFLYTEK (\textbf{IFLY.}), short text classification \textbf{TNEWS},
semantic similarity \textbf{AFQMC}, coreference resolution (CoRE) CLUEWSC 2020 (\textbf{WSC.}), and key word recognition (KwRE) \textbf{CSL}.

\textbf{CLUE MRC}:
Span selection based MRC CMRC 2018 (\textbf{CMRC}), multiple choice questions \textbf{C$^3$}, and idiom cloze \textbf{ChID}.

\textbf{Sequence Labeling}:
Chinese word segmentation (CWS) \textbf{MSR} dataset from SIGHAN2005 \cite{emerson2005second}, and
named entity recognition (NER) \textbf{MSRA-NER} \cite{levow2006third}.

We probe our proposed Lattice-BERT model thoroughly with these various downstream tasks.
The statistics and hyper-parameters of each task are elaborated in Appendix~\ref{app:data}.
We tune learning rates on validation sets and report test results with the best developing performances for CLUE tasks.\footnote{We report accuracy / exact-match scores for CLUE tasks, and label-F1 / F1 for NER / CWS tasks. CLUE results see \url{https://www.cluebenchmarks.com/rank.html}.}
For MSR and MSRA-NER, we run the settings with the best learning rates five times and report the \textbf{\textit{average scores}} to ensure the reliability of results.

\subsection{Compared Systems}

\begin{table*}[h!]
\small
\centering
\setlength{\tabcolsep}{4.5pt}
\begin{tabular}{l cccccc|c|ccc|c|cc|c}
\toprule
\multirow{3}{*}{Task} & \multicolumn{7}{c|}{CLUE-Classification} & \multicolumn{4}{c|}{CLUE-MRC} & \multicolumn{2}{c|}{\scriptsize Seq. Labeling} & \multirow{3}{*}{\shortstack{\\avg.}}  \\ \cmidrule{2-14}
 &
  NLI &
  \multicolumn{2}{c}{TC} &
  \multicolumn{1}{c}{SPM} &
  \multicolumn{1}{c}{CoRE} &
  \multicolumn{1}{c|}{KwRE} &
  \multirow{2}{*}{ \shortstack{\scriptsize avg.}} &
  \multicolumn{3}{c|}{MRC} &
  \multirow{2}{*}{ \shortstack{\scriptsize avg.}} &
  CWS &
  \multicolumn{1}{c|}{NER} &
   \\
  \cmidrule{2-7}
  \cmidrule{9-11}
  \cmidrule{13-14}
 &
  \multicolumn{1}{c}{\scriptsize CMNLI} &
  \scriptsize TNEWS &
  \multicolumn{1}{c}{\scriptsize IFLY.} &
  \scriptsize AFQMC &
  \multicolumn{1}{c}{\scriptsize WSC.} &
  \multicolumn{1}{c|}{\scriptsize CSL} & &
  \scriptsize CMRC &
  \scriptsize ChID &
  \scriptsize C$^3$ & &
  MSR &
  \multicolumn{1}{c|}{\scriptsize MSRA}
   \\ \midrule
\multicolumn{15}{c}{\textit{base}-size settings} \\
\midrule
 RoBERTa &
  80.5 &
  67.6 &
  60.3 &
  74.0 &
  76.9 &
  84.7 &
  74.0 &
  \textbf{75.2} &
  83.6 &
  66.5 &
  75.1 &
  98.2 &
  96.8 &
  78.5 \\
NEZHA &
  81.1 &
  67.4 &
  59.5 &
  74.5 &
  - &
  83.7 &
  - &
  72.2 &
  84.4 &
  71.8 &
  76.1 &
  - &
  - &
  - \\
BERT-word &
  80.0 &
  68.2 &
  60.0 &
  73.5 &
  75.5 &
  85.2 &
  73.7 &
  41.3 &
  80.9 &
  67.0 &
  63.1 &
  - &
  - &
  - \\
AMBERT &
  \textbf{81.9} &
  \textbf{68.6} &
  59.7 &
  73.9 &
  78.3 &
  \textbf{85.7} &
  74.7 &
  73.3 &
  \textbf{86.6} &
  69.6 &
  76.5 &
  - &
  - &
  - \\ \midrule
 BERT-Our &
  80.3 &
  67.7 &
  62.2 &
  74.0 &
  79.3 &
  81.6 &
  74.2 &
  72.7 &
  84.1 &
  68.6 &
  75.1 &
  98.4 &
  96.5 &
  78.7 \\
LBERT &
  81.1 &
  68.4 &
  \textbf{62.9} &
  \textbf{74.8} &
  \textbf{82.4} &
  84.0 &
  \textbf{75.6} &
  74.0 &
  \textbf{86.6} &
  \textbf{72.7} &
  \textbf{77.8} &
  \textbf{98.6} &
  \textbf{97.1} &
  \textbf{80.2}  \\
  \midrule
\multicolumn{15}{c}{\textit{lite}-size settings} \\
\midrule
 BERT-Our &
  77.9 &
  66.7 &
  60.7 &
  72.1 &
  62.4 &
  78.7 &
  69.7 &
  68.3 &
  78.7 &
  61.6 &
  69.5 &
  98.1 &
  95.5 &
  74.6  \\
 LBERT &
  79.1 &
  68.2 &
  61.9 &
  72.4 &
  70.0 &
  81.9 &
  72.3 &
  69.9 &
  81.3 &
  63.6 &
  71.6 &
  98.4 &
  96.2 &
  76.6  \\
\bottomrule
\end{tabular}
\caption{\label{Table-general}
The results on testing sets of 11 Chinese tasks. The \textbf{bold numbers} are the best scores in each column.
}
\end{table*}

\textbf{RoBERTa} \cite{cui2020revisiting} is the Chinese version RoBERTa model \cite{roberta}, which adopts the whole word masking trick and external pre-training corpus, known as RoBERTa-wwm-ext.\footnote{\url{https://huggingface.co/hfl/chinese-roberta-wwm-ext}} \\
\textbf{NEZHA} \cite{wei2019nezha} is one of the best Chinese PLMs with a bag of tricks, which also explores attention-level position embedding. \\
\textbf{AMBERT} \cite{zhang2020ambert} is the state-of-the-art multi-granularity Chinese PLM, with two separated encoders for words and characters. \\
\textbf{BERT-word} is a Chinese PLM baseline, taking words as single-granularity inputs. We obtain the results from \citet{zhang2020ambert} directly. \\
\textbf{BERT-our} is our implemented BERT model, with the same pre-training corpus, model structures, hyper-parameters, and training procedure with Lattice-BERT, but taking characters as inputs. We also adopt the whole word masking trick. \\
\textbf{LBERT} is our proposed Lattice-BERT model, with word lattices as inputs, equipping with \textit{lattice position attentions} and \textit{masked segment prediction}.

\subsection{Main Results}

In Table~\ref{Table-general},
we can see in text classification, MRC, and sequence labeling tasks, with both \textit{base} and \textit{lite} sizes, LBERT works better than our character-level baselines consistently.
LBERT-\textit{base} outperforms all previous \textit{base}-size PLMs in average scores and obtain the best performances in 7 of the 11 tasks.

Comparing with the mono-granularity PLMs in \textit{base}-size, LBERT takes benefits from word-level information and outperforms its character-level counterpart, BERT-our, by 1.5\% averagely.
Meanwhile, LBERT performs better than the word-level model, BERT-word, remarkably on CLUE tasks.
We think the lattice inputs incorporate coarse-grained semantics while avoiding segmentation errors by combining multiple segmentation results.
Therefore, with the multi-granularity treatments in word lattices, PLMs obtain better performances in downstream tasks than the mono-granularity settings.

Furthermore, LBERT outperforms the previous state-of-the-art (\textit{sota}) multi-granularity PLM, AMBERT \cite{zhang2020ambert}, by 0.9\% in text classification and 1.3\% in MRC, averagely.
Different from modeling the characters and words separately, the graph representations of word lattices could enhance the interaction between multi-granularity tokens and utilize all possible segmentation results simultaneously.
As a result, LBERT achieves a new \textit{sota} among the \textit{base}-size models on the CLUE leaderboard as well as the sub-leaderboards for text classification and MRC tasks.\footnote{\url{https://www.cluebenchmarks.com/rank.html}, the \textit{sota} by the time of submission, Oct. 31st, 2020.}

With \textit{lite}-size settings, LBERT brings 2.0\% improvement over BERT-our on average, which is larger than the case in \textit{base}-size.
In CWS, TNEWS, and CSL, the \textit{lite}-size LBERT even outperforms the \textit{base}-size BERT-our.
With more coarse-grained inputs, the shallower architectures do not require complicated interactions to identify character combinations but utilizing word representations explicitly,
thus, narrowing the gap with the deeper ones.

\subsection{Analysis}

\paragraph{Ablation Study.} We conduct ablation experiments to investigate the effectiveness of our proposed \textit{lattice position attention} (LPA) and \textit{masked segment prediction} (MSP) in downstream tasks.
To reduce the computational costs, we base our pre-training settings on \textit{lite}-size with the sequence length of 128 characters.
We select one task from each of the task clusters. We use the entity-level F1 score for NER to highlight the influence on boundary prediction.
We report the \textit{\textbf{average scores}} over 5 runs and use the development sets for CLUE tasks.

\begin{table}[t]
\small
\centering
\setlength{\tabcolsep}{5pt}
\begin{tabular}{l cccc}
\toprule
& WSC. & NER.~(EF1) & CMRC & avg. \\\midrule
BERT-our &
66.3{\tiny~(1.9)}&
92.8{\tiny~(0.2)} &
57.2{\tiny~(0.7)} &
72.1 \\
LBERT &
75.3{\tiny~(1.3)} &
94.1{\tiny~(0.1)} &
64.5{\tiny~(0.5)} &
78.0\\
\quad --Rel. &
75.7{\tiny~(1.2)} &
93.7{\tiny~(0.2)} &
63.8{\tiny~(0.8)} &
77.7\\
\quad --Dis. &
73.8{\tiny~(0.7)} &
93.8{\tiny~(0.2)} &
63.1{\tiny~(0.4)} &
76.9\\
\quad --Dis. --Rel. &
72.8{\tiny~(0.5)} &
93.6{\tiny~(0.1)} &
61.7{\tiny~(0.4)} &
76.0\\
\quad --MSP &
72.2{\tiny~(1.0)} &
93.9{\tiny~(0.1)} &
63.0{\tiny~(0.7)} &
76.4\\
\bottomrule
\end{tabular}
\caption{\label{Table-ablation}
Ablation results in \textit{lite}-size settings with standard deviation in subscripts.
EF1 is the entity-level F1 score.
-Dis. and -Rel. represent the ablation of the relative distances and positional relations in LPA, respectively.
The small numbers in brackets are standard deviation scores over five runs.
}
\end{table}

We can see in Table~\ref{Table-ablation} that the ablation of either module (--Dis.--Rel. \& --MSP) leads to a substantial drop in the average scores.
In particular, replacing MSP with vanilla MLM, the average score of --MSP drops by 1.6\%.
For the WSC. task, where long-range dependency is required to resolve the coreference, the gap is high up to 3.1\%.
We trace this drop into the pre-training procedure and observe the MLM accuracy for the --MSP setting on the development set is 88.3\%.
However, if we mask the tokens within the segment and avoid potential leakages, the accuracy drastically drops to 48.8\%, much lower than the performance of LBERT training with MSP (56.6\%).
This gap provides evidence that the MSP task prevents the PLMs from tricking the target by
peeking the overlapping text units in one segment, thus encourages the PLMs to characterize the long-range dependency.

For the LPA method, without the positional relation (--Rel.), the entity-level F1 score on NER decreases by 0.4\%, and the performance on CMRC decreases by 0.7\%.
The performance drops are similar to the case without distance information (--Dis.).
Without either of them (--Dis. --Rel.), the gaps widen to 0.5\% and 2.8\%, respectively.
The boundary predictions in NER and CMRC are more sensitive to the local linguistic structures like nested words or overlapping ambiguity.
With the positional relation and distance charaterized in attention, LBERT could accurately model the interaction between the nested and overlapping tokens in different segmentation results.
Meanwhile, the accuracy of WSC. remarkably drops without distance information.
The performance drops by 7.5\% and 5.8\% when the number of characters between the pronouns and candidate phrases is larger than 30, or between 20 to 30, respectively.
For the rest cases, the drop is only 0.4\%.
With explicitly modeling of distance, LBERT predicts the long-distance coreference relations more accurately.
Averagely, without the positional relation and distance modeling in LPA, the performance drops by 2.0\% on the three tasks, showing the importance of LPA in assisting the PLMs to exploit the multi-granularity structures in word lattices.

\paragraph{How LBERT Improves Fine-grained PLMs?}
We compare the prediction results of LBERT and the character-level BERT-our in \textit{base}-size on development sets to investigate how the LBERT outperforms the vanilla fine-grained PLMs.
Intuitively, the word-level tokens in lattices provide coarse-grained semantics, which argument the character-level inputs.

We observe in TNEWS, the short text classification task, LBERT brings more improvement in the shorter instances, where the statements may be too short to provide enough context for predictions.
By dividing the development set into five bins with equal size according to the sentence length, LBERT outperforms BERT-our by 2.3\% and 1.3\% in the shortest and second shortest bins, respectively,
larger than the average gain on the rest instances (0.6\%).
We think the redundant tokens in word lattices provide rich context for the semantics of these short statements.
For example, for the short title 我们村的电影院/\textit{the cinema in our village}, with the redundant words, 电影/\textit{movie}, 影院/{cinema}, and 电影院/\textit{cinema}, introduced in the lattice, LBERT classifies the instance as entertainment news instead of news stories.

Another case is the CSL task, where the target is to predict whether the candidate words are keywords for a given paragraph.
For those instances, where LBERT
identifies more than two word-level tokens from each candidate word averagely,
which accounts for 47\% of the dataset, the performance gain is 3.0\%, significantly larger than the average improvement of the rest, 1.0\%.
We think LBERT understands the key words from various aspects by exploiting the redundant expressions in lattices.
For example, from the keyword candidate 太阳能电池/\textit{solar battery}, the 太阳/{solar}, 太阳能/{solar energy}, 电池/{battery}, and 太阳能电池/\textit{solar battery} are lattice tokens.
With these word-level tokens, LBERT could match this candidate with the expressions in the paragraph like 阳极/\textit{positive electrode}, 光/\textit{light}, 电子/\textit{electron}, 离子/\textit{ion}, etc.

On the other side, for MSRA-NER,
LBERT reduces the errors in identifying entities with nested structures.
Averagely, the number of error cases where the predicted entities are nested with the golden ones are reduced by 25\% in LBERT.
For example, the organization entity 解放巴勒斯坦运动/\textit{Palestine National Liberation Movement} is nested with the location entity 巴勒斯坦/\textit{Palestine} and ends with an indicator to organizations, 运动/\textit{movement}.
The character-level baseline model mistakenly recognizes the 巴勒斯坦/\textit{Palestine} and 动/\textit{move} as a location and an organization, separately.
While LBERT identifies this entity correctly after integrating the words, 解放/\textit{liberate}, 巴勒斯坦/\textit{Palestine}, and 运动/\textit{movement}.
With the pre-trained multi-granularity representations, LBERT fuse the contextual information from words and characters simultaneously, and detects the correct entity in success.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.51\textwidth]{attention2.pdf}
\caption{Visualization of the attention scores of  研究生活很充实/\textit{Research life is very fulfilling}.
The three lines from top to bottom are the attentions before fine-tuning and after fine-tuning with MSRA-NER and TNEWS tasks, respectively.
\label{fig:att-diff}
}
\end{figure}

\paragraph{How does LBERT harness multi-Granularity representations?}
LBERT consumes all the words and characters from input sequences simultaneously, but how does the model utilizes such multi-granularity representations during pre-training and downstream tasks?
To investigate this, we use the average attention scores that each lattice token receives among all layers and all heads to represent its importance.

As the example shown in Figure~\ref{fig:att-diff},
before fine-tuning, LBERT focuses on tokens including
活/\textit{live}, 充实/ \textit{fulfilling}, 研究/\textit{research},  研究生/\textit{graduate student}, 究/\textit{investigate}, etc.
Before fine-tuning on specific tasks, the model captures various aspects of the sentence.
After fine-tuning with MSRA-NER, the most focused words become 充实/\textit{fulfilling}, 很/\textit{very}, 生活/\textit{life}, and 研究/\textit{research}, i.e., the tokens from the golden segmentation result, ``研究|生活|很|充实'', which is intuitively beneficial for the NER tasks.
The attention score of the wrong segmented word, 研究生/\textit{graduate student}, drops remarkably.

On the other hand, after fine-tuning with the news title classification task, TNEWS, LBERT tends to focus on 充实/\textit{fulfilling}, 研究生/\textit{graduate student}, 生活/\textit{life}, etc.
Although these tokens can not co-exist in one Chinese word segmentation result, LBERT can still utilize the redundant information from various plausible segmentations to identify the topics of inputs.
These results indicate that Lattice-BERT can well manage the lattice inputs by shifting the attention to different aspects among the multi-granularity representations according to specific downstream tasks.

\begin{table}[]
\setlength{\tabcolsep}{4pt}
\small
\centering
\begin{tabular}{p{7.5cm}}
\toprule
\textbf{Question}: What is the game with the theme song which is sung by Chen Yiting and is composed by Zeng Zhihao? \\ \midrule
\textbf{Document}: $\cdots$ He was recommended by Lu Shengfei, the No.7 music director of the Cape, as the composer of the theme song \textbf{\textit{The South of China}}$_{\left[\textrm{err}\right]}$. $\cdots$
In cooperation with singer Chen Yiting, Zeng Zhihao was responsible for the composition of the theme song Wish and the promotion song Youth Love for \textbf{\textit{The Legend of Sword and Fairy V}}$_{\left[\textrm{ans}\right]}$. \\
\bottomrule
\end{tabular}
\caption{ An example in CMRC. Given the \textbf{Question}, LBERT predicts the \textbf{\textit{corrected answer}}$_{\left[\textrm{ans}\right]}$ in the \textbf{Document}, while BERT-our predicts the \textbf{\textit{wrong answer}}$_{\left[\textrm{err}\right]}$.
\label{tab:case_study}}
\end{table}

\paragraph{Case Study.}
Table~\ref{tab:case_study} shows an example in CMRC, a span selection MRC task, where models choose a text span from the given document to answer the question.
In this case, the question asks for a \textit{game}, restricted by its \textit{theme song}.
BERT-our incorrectly outputs a theme song, \textit{The Song of China}, since there is no expression in the document explicitly related to \textit{game}.
However, LBERT find the correct answer, \textit{The Legend of Sword and Fairy V}.
One possible reason is that \textit{The Legend of Sword and Fairy} is an entry in the vocabulary for lattice construction.
LBERT may have learned this word as an entity for a famous video game from the context in pre-training by explicitly exploiting its representation as a whole.
With the coarse-grain text units in pre-training, LBERT directly encodes knowledge about these units to benefit the downstream tasks.

\paragraph{Computational Costs. }
For fair comparisons, we ensure LBERT and the character-level baselines (i.e., BERT-our) have the same training epochs when the training steps are equal following previous works \cite{zen,zhang2020ambert}.
Thus, comparing with BERT-our, 35\% more text units are introduced in the pre-training instances of LBERT, which introduces 48\% more computational resources comparing with BERT-our to process the additional word-level tokens (See Appendix~\ref{app:imp}).
To illustrate the gains attribute to the incorporation of lattices instead of additional computations, we investigate the \textit{lite}-size BERT-our with longer input sequences in pre-training, which has the same computational costs as LBERT.
We find LBERT still outperforms BERT-our by 2.2\% averagely on CLUE classification tasks.
More details are elaborated in Appendix~\ref{app:exp}.

\section{Related Works}

Recently, several works utilize the lattice structures to explore multi-granularity information in Chinese NLU tasks.
\citet{buckman-neubig-2018-neural} incorporate lattice into recurrent neural networks based language modeling to capture marginal possibilities across all possible paths.
In NER, Lattice-LSTM \cite{latticeLSTMner}, graph neural networks \cite{gui2019lexicon}, and flat-lattice transformers \cite{flat} are adopted to incorporate words from lattice inputs.
\citet{lai2019lattice} adapt convolutional neural networks to lattice for matching based question answering.
\citet{chen-etal-2020-neural} adopt graph matching networks to perform multi-granularity interaction between lattices for text similarity.
These works are designed to explore word lattices in specific tasks.
We explore the multi-granularity representations with word lattices in PLMs, investigating the previously attempted downstream tasks as well as other tasks, e.g., MRC.
We design LPA to meet various interaction needs of the downstream tasks and propose MSP to avoid the leakages.

In the field of Chinese PLMs, some efforts incorporate coarse-grained information with the character-level inputs.
ERNIE 1.0 \cite{ernie1.0} and BERT-wwm \cite{bert-wwm} propose to mask the words, entities, and phrases as a whole in the MLM task to encourage the modeling of coarse-grained features.
ZEN \cite{zen} adopt auxiliary networks to integrate n-gram representations.
BERT-MWA \cite{bertmwa} propose word-aligned attention to use multiple segmentation boundaries.
Different from their methods, we propose Lattice-BERT to consume multi-granularity tokens simultaneously into one PLM via lattice graphs.
Thus, Lattice-BERT explicitly exploits the representations of the coarse-grained units, as well as the interactions among word- and character-level tokens.
The proposed MSP task can be treated as an extension of the whole word masking \cite{bert-wwm}, while considering the span information like \citet[SpanBERT]{spanbert} according the lattice structures.
The concurrent work, \citet[AMBERT]{zhang2020ambert} investigate multi-granularity inputs similarly, but they use two transformer encoders to separately deal the word and character sequences.
We treat the words and characters as lattice graphs, which enables thorough interactions among multi-granularity tokens and utilizes all potential segmentation results.

\section{Conclusions}
In this paper, we propose Lattice-BERT to leverage multi-granularity representations of input sentences for Chinese PLMs.
Specifically, Lattice-BERT takes a word-lattice as input, modeling the representations of words and characters simultaneously.
We design the \textit{lattice position attention} to embed the multi-granularity structure into transformers and propose the \textit{masked segment prediction} task to avoid potential leakage in original MLM caused by the redundancy information in lattices.
We conduct extensive experiments on 11 Chinese NLU tasks and observe consistent gains over character-level baselines, achieving new \textit{sota} on CLUE benchmarks.
We show that Lattice-BERT can well manage the lattice inputs and utilize multi-granularity representations to augment the character-level inputs.
We believe the lattice structure can be adapted to integrate the phrase and word representations into the word-piece based PLMs in other languages, which we leave for future exploration.

\section*{Acknowledgments}
This work is supported in part by the National Hi-Tech RD Program of China (No.
2018YFB1005100), the NSFC under grant agreements (61672057, 61672058). For any correspondence, please contact Yansong Feng.

\newpage


\section{Introduction}
Emotion cause extraction was firstly defined by ~\cite{Rule-AText-drivenRule-basedSystemforEmotionCauseDetection}, aiming at extracting possible causes in documents with a given emotion. Previously, ECE has aroused a growing number of researches, including rule-based methods ~\cite{Rule-EMOCause,Rule-EmotionCauseDetectionwithLinguisticConstructions,Rule-ExtractingCausesofEmotionsfromText}, machine learning methods ~\cite{ML-EmotionCauseDetectionwithLinguisticConstructioninChineseWeiboText,SVM-Event-DrivenEmotionCauseExtractionwithCorpusConstruction} and neural networks ~\cite{SVM-LSTM-AnEmotionCauseCorpusforChineseMicroblogswithMultiple-UserStructures,Context-awareemotioncauseanalysiswithmulti-attention-basedneuralnetwork,ACo-AttentionNeuralNetworkModelforEmotionCauseAnalysiswithEmotionalContextAwareness,Context-AwareMulti-ViewAttentionNetworksforEmotionCauseExtraction,MultipleLevelHierarchicalNetwork-BasedClauseSelectionforEmotionCauseExtraction,ARNN-TransformerHierarchicalNetworkforEmotionCauseExtraction,JointLearningforEmotionClassificationandEmotionCauseDetection,ExtractingEmotionCausesUsingLearningtoRankMethodsFromanInformationRetrievalPerspective,AdversarialTrainingbasedCross-lingualEmotionCauseExtraction,AKnowledgeRegularizedHierarchicalApproachforEmotionCauseAnalysis}. However, such approaches have a high dependency on labelled emotions. ~\cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts} experimented with a removal of annotated emotions on ECE methods, and the F1 score drops dramatically about 34.69\% \cite{ACo-AttentionNeuralNetworkModelforEmotionCauseAnalysiswithEmotionalContextAwareness}.

\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{pic1.png}
\caption{An intuitive example of differences between ECE and ECPE task.}
\label{diff}
\end{figure}

Expensive labelling limit the application of ECE-driven techniques. To address the limitation, recent works focused on the emotion-cause pair extraction (ECPE) task \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}, which aims to extract pairs of emotions and their corresponding causes in a document. Figure~\ref{diff} is an intuitive example of differences between ECE and ECPE task. Compared with ECE task, ECPE task is more challenging as it requires no transcendental information of annotated emotions and extracts emotions, causes and emotion-cause pairs from plain text. Existing approaches follow a two-step pipeline \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}: detecting emotions and causes firstly and then pairing them from a Cartesian product of emotions and causes. However, these methods overlook the error propagation from the first stage to the second. Intuitively, when an emotion or cause is predicted as negative in step one, it will not participate in step two and completely becomes negative. Table~\ref{study1} presents the recall rate of two successive steps. The recall rate of the second step is highly correlated to that of the first step. In addition, \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts} ignores contextual and semantic information when pairing emotions and causes, which is essential for ECPE. For example, a student would probably be happy when he heard he got an A in a class, but would also possibly be sad if his goal was to pass all the courses with an A+. In this way, a common emotion-cause pair can be a wrong pair given a particular context.

\begin{table}[h]
\caption{Results of recall rate of three sub-tasks in ECPE task generated by three models \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}. More details are given in Section~\ref{Baselines}}
\centering
\begin{tabular}{l|cc|c}
\hline
 & \multicolumn{2}{c|}{Step1} & Step2 \\ \hline
 & \tabincell{c}{Emotion\\ Extraction}  & \tabincell{c}{Cause\\ Extraction}  & \tabincell{c}{Emotion-cause\\ Pair Extraction}\\ \hline
Indep & 0.8071 & 0.5673  & 0.5082 \\
Inter-CE & 0.8122 & 0.5634  & 0.5135 \\
Inter-EC & 0.8107 & 0.6083 & 0.5705 \\
\hline
\end{tabular}
\label{study1}
\end{table}

In order to tackle these problems, we propose a model called Dual-Questioning Attention Network (DQAN). Specifically, weighted loss function is applied to address error propagation and better focus on imbalanced text. We introduce a hierarchical network with two levels including word level and clause level. A novel dual-questioning mechanism is applied to jointly consider the representation of candidate pairs of emotion and cause and the rest of the context. Features of candidate emotions and causes are extracted and matched through the contextual information for fully utilization. Furthermore, we propose a method to incorporate and encode distance information in pair extraction. To evaluate the effectiveness of our model, we conduct experiments on benchmark ECPE dataset and ECE datasets. The main contributions of our work can be summarized as follows:
\begin{itemize}
\item We propose a dual-questioning attention network to handle the ECPE task in a unified context, which learns useful feature for pairing and matching across the semantics.
\item We employ a weighted loss function for the imbalanced task, which alleviate and moderate error propagation appeared in previous works.
\item We conduct experiments on benchmark datasets to show the effectiveness of our model and the performance shows that our model significantly outperforms the baseline models.
\end{itemize}

The rest of this paper is organized as follows. Section~\ref{Related work} presents related work in emotion cause pair extraction and Section~\ref{Model} introduces our model. Experiments set up and results are reported in Section~\ref{Experiments}, followed by conclusions in Section~\ref{Conclusions}.

\section{Related work}\label{Related work}
Lee et al.~\cite{Rule-AText-drivenRule-basedSystemforEmotionCauseDetection} firstly restrict the definition of automatic emotion cause extraction. They develop two sets of rules for emotion cause detection and propose an evaluation scheme. Rule based methods have been improved by later researchers. Russo et al.\cite{Rule-EMOCause} proposed a rule based on relevant linguistic patterns and an incremental repository of common sense knowledge on emotional states. Chen et al.\cite{Rule-EmotionCauseDetectionwithLinguisticConstructions} created two sets of linguistic patterns. Syntactic and dependency parser and rules were added up to the principle by~\cite{Rule-ExtractingCausesofEmotionsfromText}. Other than rule based methods, there were also some machine learning based models. Gui et al.\cite{ML-EmotionCauseDetectionwithLinguisticConstructioninChineseWeiboText} employed SVMs and CRFs on extraction causes. Gui et al.\cite{SVM-Event-DrivenEmotionCauseExtractionwithCorpusConstruction} applied multi-kernel SVM on emotion-cause identification. They also presented an open-source dataset using SINA city news, which led to next period of prospectives in emotion cause extraction task.

Deep learning networks have been proved efficient on other NLP tasks such as sentence classification and name entity identification. Therefore, a variety of neural networks have been applied to the task. One approach to emotion cause detection is to treat it as clause classification problem. With the huge success of hierarchical attention network (HAN) \cite{HAN} on document classification, hierarchical models have been widely used on various kinds of NLP task. Li et al.\cite{Context-awareemotioncauseanalysiswithmulti-attention-basedneuralnetwork} and Li et al.\cite{ACo-AttentionNeuralNetworkModelforEmotionCauseAnalysiswithEmotionalContextAwareness} both proposed a hierarchical model, which encoded clauses with bi-directional long short term memory and applied attention mechanism to the emotion clause and applicant cause clauses. However, they simply input only one applicant cause clause at one time, which ignored information in the rest of the article and relationships between them. Other approach is to apply sequence model to extract cause clause. Cheng et al.\cite{SVM-LSTM-AnEmotionCauseCorpusforChineseMicroblogswithMultiple-UserStructures} proposed a basic long short-term memory to the task. Gui et al.\cite{AQuestionAnsweringApproachtoEmotionCauseExtraction} presented a convolutional Multiple-Slot deep memory network. Other researchers mostly presented a hierarchical model, that bi-directional long short-terms memory was often applied on encoding clauses and differences appeared in clause-level network. Xiao et al.\cite{Context-AwareMulti-ViewAttentionNetworksforEmotionCauseExtraction} proposed a multi-view attention network (COMV) and Yu et al.\cite{MultipleLevelHierarchicalNetwork-BasedClauseSelectionforEmotionCauseExtraction} applied attention mechanism on word-level and convolutional network on phrase-level. Xia et al.\cite{ARNN-TransformerHierarchicalNetworkforEmotionCauseExtraction} presented a joint emotion cause extraction framework, called RNN-Transformer Hierarchical Network (RTHN), to efficiently capture causality between clauses.

Xia et al.\cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts} pointed out the limitation of previous research on ECE and defined the task of emotion-cause pair extraction (ECPE). In their approaches, pair extraction follows a pipeline framework, including extracting candidate emotion and cause clauses and classifying the relations between them subsequently. They proposed three different models, which extracting both clauses independently or enhancing one's extraction through another. However, they ignored error propagation, which is a commonplace in pipeline models. Besides, contextual information is also ignored in their work \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}, which is necessary for pair extraction. Our model is proposed to address the mentioned problems. To be more specific, our model learns the contextual information through a novel structure, namely dual-questioning attention and shares the contextual information with candidate emotions and causes. In addition, we employ a weighted loss function to alleviate error propagation and develop a method to joint distance information with the whole context.

\section{Methodology}\label{Model}

This section introduces the overall and specific architecture of our model. Our method consists of two stages: extracting possible emotions and causes, and then pairing them.

\subsection{Task definition}
Suppose we have a document $D$, containing several clauses as $d=\{c_1,...,c_i,...,c_{|d|}\}$, where $c_i$ is the $i$-th clause in $d$. Each clause $c_i$ consists of $|c_i|$ words, as $c_i = \{w_{i}^1,...w_{i}^t,...w_{i}^{|c_i|}\}$. To prevent confusion, we apply a superscript 'e' on true emotion clause, and 'c' on true cause clause. In this work, our goal is to identify whether a clause pair $(c_i, c_j) (i, j=1,2,...|d|)$ is the true emotion-cause pair $(c^e, c^c)$ in the passage. Therefore, we can form ECPE task as a binary classification task:
\begin{equation}
f(c_i, c_j) =
\left\{
\begin{aligned}
& 1;\quad  if \; c_i  = c^c \; and \; c_j = c^e \\
& 0;\quad  otherwise
\end{aligned}
\right.
\end{equation}

\subsection{Overall architecture}
The overall architecture of our work is shown in Figure~\ref{model}, which consists of two steps. The first step aims to extract a set of emotion clauses and a set of cause clauses in each document respectively and the second step yields true pairs based on candidate emotion and cause clauses extracted from step one and semantic contexts.
\begin{figure*}[ht]
\centering
\includegraphics[width=\textwidth]{pic2.png}
\caption{Model architecture}
\label{model}
\end{figure*}

\subsection{Step1: Emotion and cause extraction}
In step one, we apply a dual-layer sequential network to jointly extract emotion and cause clauses. Considered the relationship between emotions and its causes, we extract emotion clauses on the lower layer firstly and cause clauses on the upper layer subsequently. Given a clause $c_i = \{w_{i}^1,...w_{i}^t,...w_{i}^{|c_i|}\}$, we firstly embeds words with embedding matrix $W_e$ to get an input sequence word embedding $x_{i}^t$. Then a bi-directional LSTM is applied to yield contextual representation of a word and an attention mechanism is utilized to generate a clause representation $c_i$.
\begin{align}
& h_i^t = {\rm BiLSTM}(x_i^t) \\
& {\alpha}_{i}^t =\dfrac{{\rm exp}({h_{i}^t}^Tu_w)}{\sum_t {\rm exp}({h_{i}^t}^Tu_w)} \\
& c_i = \sum_t {\alpha}_{i}^th_{i}^t
\end{align}
where $u_w$ is a learnable weight parameter.

Secondly, we construct a dual-layer network with bi-LSTMs, linear projections and softmax layer for emotion and cause extraction based on clause representations $c_i$. In addition, considered the enhancement of cause extraction with emotion extracted, we firstly detect emotion clauses and then after extract cause clauses.
\begin{align}
& \hat{y_i^e} = {\rm softmax}(W_c {\rm BiLSTM}(c_i)) \\
& \hat{y_i^c} = {\rm softmax}(W_e {\rm BiLSTM}(c_i \oplus \hat{y_i^e}))
\end{align}
where $W_c, W_e$ represents projection matrices.

Since extraction task is heavily imbalanced, we apply a weighted loss with two components for training procedure.
\begin{equation}
L =  \sum_{i}\beta_1(y_{i}^c \log(\hat{y_i}^{c}) + y_{i}^e \log(\hat{y_i}^{e}))
\end{equation}
where $y_i^c, y_i^e$ is the ground-truth of $\hat{y}_i^c, \hat{y}_i^e$, and $\beta_1$ is the weight vector of weighted loss.

\subsection{Step 2: Pair matching}

\paragraph{\bf Clause encoder}
Word-level encoder aims to obtain a clause representation $\hat{c}_i$ from sequence word vectors.
Given a clause $c_i = \{w_{i}^1,...w_{i}^t,...w_{i}^{|c_i|}\}$, we first embed the words with embedding matrix $W_e$ to get an input sequence word embedding $x_{i}^t$ and its hidden representation $h_{i}^t$ with a linear projection.
\begin{equation}
h_{i}^t = W_hx_{i}^t + b_h
\end{equation}

When generating clause representations, not all text words make the same contribution. The attention mechanism produces a context vector by focusing on different portions of the word sequence and aggregating the hidden representations of those representative words. Specifically, the weight ${\alpha}_{it}$ is computed as follows with the attention mechanism:
\begin{align}
u_{i}^t &= {\rm tanh}\left(W_wh_{i}^t+b_w\right) \\
{\alpha}_{i}^t &=\dfrac{{\rm exp}({u_{i}^t}^Tu_w)}{\sum_t {\rm exp}({u_{i}^t}^Tu_w)}
\end{align}

where $W_w, b_w, u_w$ are weight parameters and $h_{i}^t$ is the current hidden representation of the $t$-th word in clause $c_i$. Then, the weight ${\alpha}_{i}^t$ is combined with hidden representations $h_{i}^t$ to get the clause representation $\hat{c}_i $ as follows:
\begin{equation}
\hat{c}_i = \sum_t {\alpha}_{i}^th_{i}^t
\end{equation}

\paragraph{\bf Relation embedding}
Relation embedding was often applied to describe the relative position of a word from two named entities in relation extraction tasks \cite{RelationExtractionPCNN} and performed well. In our work, clause position embedding is utilized to compute relative distance from current clause to cause clause and emotion clause respectively. The clause representation can be computed by following:
\begin{equation}
c_i = \hat{c}_i \oplus p_{i1}\oplus p_{i2}
\end{equation}
$\hat{c}_i$ refers to raw clause representation from output of word-level layer and $p_{i1}$, $p_{i2}$ are relative positions based on cause clause and emotion clause.

\paragraph{\bf Dual-questioning attention}

Inspired by the outstanding performance in sentence encoding by ~\cite{Transformer}, we propose a dual-questioning attention to better focus on semantic understanding and pair extracting in ECPE task. Firstly, a simple attention function can be described as mapping a query and a set of key-value pairs to an output:

\begin{equation}
{\rm Att}(Q, K, V) = {\rm softmax}(\dfrac{QK^T}{\sqrt{d_k}})V
\end{equation}

where matrice $Q, K, V$ refer to the query, keys and value, and $d_k$ is the dimension of query and keys.

Then, we propose a dual-questioning attention since we question the candidate emotion and cause to the rest of the context respectively, each of which can be produced with a linear projection of several outputs of simple attention mechanism.

\begin{align}
&
\begin{aligned}
\hat{z_1} = & [{\rm Att}(c_e W_{11}^{c_e}, c W_{11}^{c}, c W_{11}^{c})  \\
&\oplus ... \oplus {\rm Att}(c_e W_{i1}^{c_e}, c W_{i1}^{c}, c W_{i1}^{c})]W^O \\
\end{aligned}
\\
&
\begin{aligned}
\hat{z_2} = & [{\rm Att}(c_c W_{12}^{c_c}, c W_{12}^{c}, c W_{12}^{c})  \\
& \oplus ... \oplus {\rm Att}(c_c W_{i2}^{c_c}, c W_{i2}^{c}, c W_{i2}^{c})]W^O \\
\end{aligned}
\end{align}

where $W_{ij}^{c} \in \mathbb{R}^{d_{c} \times d_c}$, $W_{ij}^{c_e} \in \mathbb{R}^{d_{c_e} \times d_{c_e}}$ and $W_{ij}^{c_c} \in \mathbb{R}^{d_{c_c} \times d_{c_c}}$  are projection matrices and $d_{c}, d_{c_e}, d_{c}$ refer to the dimension of the representation of a sequence of clause $c = \{c_1, c_2, ..., c_{|d|}\}$, cause clause $c_c$ and emotion clause $c_e$. The $i$ represents the number of heads that we generate outputs from the simple attention mechanism, which is a hyper-parameter that we set in the next section.

Then, the output $\hat{z_1}, \hat{z_2}$ are combined with the query of inputs and fed into a normalization layer as follows.
\begin{equation}
z = {\rm Norm}(\hat{z_1}\oplus c_e) \oplus {\rm Norm}(\hat{z_2}\oplus c_c)
\end{equation}

\paragraph{\bf Classification}
The fully connected layer is used on classification according to features produced by the normalization layer. The output is then computed with a linear transformation by following:
\begin{equation}
o = wz+b
\end{equation}
$w$ and $b$ are weighted matrix and bias vector, and output vector $o \in \mathbb{R}^2$ is the output of fully connected layer. We finally add a softmax function to compute probability distribution $\hat{y}_i$ from the output vector.
\begin{equation}
\hat{y}_i = \dfrac{\exp(o_i)}{\sum_i \exp(o_i)}
\end{equation}

\paragraph{\bf Training procedure}
For pair filtering, we have more negative candidate pairs which must lead to a more imbalanced problem. In this way, we apply a weighted loss function to address the issue, focusing on positive pairs.
\begin{equation}
L_{pair} = \sum_{c_e, c_c}(\beta_2 y_{c_e,c_c} \log(\hat{y}_{c_e,c_c})) + \lambda||\theta||^2
\end{equation}
where $y_{c_e,c_c}$ is the ground-truth of $\hat{y}_{c_e,c_c}$, and $\beta_2$ is the weight vector of weighted loss. A $l2$-normalization is also added to the loss function with $\lambda$ restricted.

\section{Experiments}\label{Experiments}

\subsection{Datasets and experimental settings}
We conducted experiments on benchmark ECPE dataset released by \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}. Table \ref{ECPE dataset} presents details of the dataset.  The
metrics we used in evaluation follows\cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}. We use precision, recall and f1-score as evaluation metrics, which are calculated as follows:
\begin{equation}
\begin{aligned}
&P = \dfrac{\sum_{correct\_pairs}}{\sum_{proposed\_pairs}} \\
&R = \dfrac{\sum_{correct\_pairs}}{\sum_{annotated\_pairs}} \\
&F1 = \dfrac{2\times P \times R}{P+R} \\
\end{aligned}
\end{equation}
where proposed pairs denotes the emotion cause pairs predicted by the model, annotated pairs denotes the emotion cause pairs that are labeled in the dataset and correct pairs means the pairs that are both labeled and predicted as an emotion cause pair.

\begin{table}[h]
\centering
\caption{Details of the ECPE dataset.}
\label{ECPE dataset}
\begin{tabular}{l|c|c} \hline
 & Number & Percentage \\ \hline
Documents & 1945 & 100\% \\
Documents with 1 emotion-cause pair & 1746 & 89.77\% \\
Documents with 2 emotion-cause pairs & 177 & 9.10\% \\
Documents with $\ge$ 2 emotion-cause pairs & 22 & 1.13\% \\ \hline
\end{tabular}
\end{table}

The word vectors \cite{word2vec} are pre-trained on the corpora from Chinese Weibo by word2vec and keep unchanged during the training duration. The dimension of word embedding and relative position embedding are set to be 200 and 50 respectively. The number of heads in multi-questioning attention is set to 4. In order to avoid overfitting, we apply dropout to embeddings and inputs of softmax layer, which is set to 0.5 and 0.8. Besides, L2-normalization is set as 1e-5. For training details, we use the Adam optimization with shuffled batches. Batch size and learning rate are set to 128 and 0.005. The best weight parameters in stage one and two are 3 and 2.5, which will be discussed in next section. Furthermore, all weight matrices and bias are randomly initialized by a uniform distribution U (-0.01, 0.01).

\subsection{Baselines}\label{Baselines}
We compare our approach with the following baselines:
\begin{itemize}
\item {\bf Indep} independently extracts emotions and causes by two Bi-LSTMs and then pairs the extracted emotions and causes with a simple classifier \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}.
\item {\bf Inter-CE} generally follows the procedure of Indep, however, enhances emotion extraction with the predictions of cause extraction \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}.
\item {\bf Inter-EC} generally follows the procedure of Indep, however, enhances cause extraction with the predictions of emotion extraction \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}.
\item {\bf Indep-W} is an extended model of {\bf Indep} with weighted loss on two stages.
\item {\bf Inter-CE-W}  is an extended model of {\bf Inter-CE} with weighted loss on two stages.
\item {\bf Inter-EC-W}  is an extended model of {\bf Inter-EC} with weighted loss on two stages.
\end{itemize}

\subsection{Results}
For the purpose of simplicity, we denote the proposed model as DQAN. We report related evaluation metrics of our method and all baselines on the test sets.
\begin{table}[h]
\centering
\caption{Experiment results of all proposed models and baselines in emotion-cause pair extraction with precision, recall, and F1-measure as metrics.}
\label{ECPE}
\begin{tabular}{l|ccc}\hline
 &  \multicolumn{3}{c}{Emotion-cause Pair Extraction} \\ \hline
Methods & Precision & Recall & F1 \\ \hline
{\bf Indep \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}} & 	0.6832&	0.5082	&	0.5818\\
{\bf Inter-CE  \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}} &	{\bf 0.6902}&	0.5135	&	0.5901\\
{\bf Inter-EC  \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}} &	0.6721&	0.5705	&	0.6128\\  \hline
{\bf Indep-W} & 	0.6155	&0.5768	&0.5940\\
{\bf Inter-CE-W} 	&0.6206	&0.5910	&0.6047\\
{\bf Inter-EC-W} & 0.6658	&0.5823	&0.6205\\ \hline
{\bf DQAN}&    0.6733& {\bf 0.6040}   &   {\bf  0.6362}\\ \hline
\end{tabular}
\end{table}

The experimental results of our method and the baselines on ECPE dataset are shown in Table~\ref{ECPE}. Our results are performed with $p<0.05$ tested by the Student’s paired t-test. Compared with {\bf Inter-EC}, {\bf DQAN} shows an improvement of 3.82\% f1-score, 5.87\% recall and 0.16\% precision in emotion-cause pair extraction. It indicates that weighted loss works well in reduction of error propagation and attention mechanism properly integrates contextual semantics.

Furthermore, in order to confirm the effectiveness of our weighted loss in alleviating error propagation by enhancing recall rate of the two steps presented in Table \ref{study1}, we experiment existing methods with our weighted loss. The approaches {\bf Inter-EC-W, Inter-CE-W, Indep-W} are extended models of {\bf Inter-EC, Inter-CE, Indep} with weighted loss on two stages. For the f1 measurement, the three derivative models achieve 1.26\%, 2.47\% and 2.10\% higher than their origin models respectively, which proves that our weighted loss is beneficial on ECPE task. Beyond that, our {\bf DQAN} outperforms 3.73\% and 2.53\% on recall rate and f1-score compared with {\bf Inter-EC-W}. The difference between {\bf Inter-EC-W} and {\bf DQAN} is that {\bf DQAN} uses a dual-questioning attention module to obtain the emotion and cause representation from context while {\bf Inter-EC-W} ignores context and simply classifies emotions and causes. The better performance on recall rate and f1-score indicates that dual-questioning attention module can effectively combine with weighted loss to push it further on alleviation of error propagation by  jointly considering contextual semantics.

\subsection{Discussion}

\subsubsection{Ablation study}
To understand the efficacy of dual-questioning attention modules and weighted loss functions, we conduct ablation experiment on these modules.

\begin{table}[h]
\centering
\caption{Ablation study for the DQAN model}
\label{ablation}
\begin{tabular}{l|c|c|c} \hline
Methods & P & R & F1 \\ \hline
DQAN & 0.6733& 0.6040   &  {\bf  0.6362}\\
DQAN w/o dual-questioning &0.6728&0.5815 &0.6234 \\
DQAN w/o weighted loss 1 &{\bf 0.7089}	&0.5503	&0.6184 \\
DQAN w/o weighted loss 2 &0.6418	&{\bf 0.6082}	&0.6234 \\
Inter-EC-W &0.6658	&0.5823	&0.6205\\ \hline
\end{tabular}
\end{table}

From Table~\ref{ablation}, we can observe that after reducing dual-questioning attention, the recall and f1-score of {\bf DQAN} drop 3.59\% and 2.47\%. In addition, {\bf DQAN} without dual-questioning attention module performs practically as same as {\bf Inter-EC-W}, which setups the same weight and ignores contextual semantics. The main reason is that dual-questioning attention is capable, enforcing the model to pay more attention on contextual semantics that can enhance the connection between emotions and causes. Besides, with a removal of weighted loss function in step one and two, the f1-score drop about 2.80\% and 2.01\% respectively, which proves our weighted loss function is effective in capturing emotion-cause pairs. Between such two weighed loss functions, weighted loss in step one is more essential on cutting error propagation. The precision is the highest when removing weighted loss one and the recall rate is the highest when removing weight loss two, mainly because the former aims to enhance and expand candidate emotion-cause pairs while the latter aims to narrow then down.

\subsubsection{Effects of weighted loss}

During the experiment, we find that the two weighted loss contributes significantly to the results. In order to explore the real influence of weighted loss, we set different weight on weighted loss and leave other parameters as they are, and observe evaluation on ECPE task.

\begin{figure}[h]
\centering
\includegraphics[width=0.49\linewidth]{weight1.png}
\includegraphics[width=0.49\linewidth]{weight2.png}
\caption{Weight variation of weighted loss in stage one and two\label{subfig:left}}
\end{figure}

Figure~\ref{subfig:left} firstly illustrates how the performance varies on different weight parameters in stage one. With the increase of the value of weight, we can see an increase of recall and a decrease of precision, and that both a small weight and a large weight will cause f1-score to drop. This is because a greater weight on positive clauses could enlarge the recall rate of emotion and cause extraction in stage one and lead to an increase of recall of emotion-cause pair extraction. But it's also a burdensome to the model of stage two, which result in a decrease of precision. Therefore, the highest f1-score appears when a proper weight is set because f1-score is a harmonic mean of precision and recall. It also shows a performance variation with varied weight  parameters in stage two, a practically mirrored picture of that in stage one. This is because we set a greater weight on negative pairs in stage two instead of positive labels in stage one. Both of the results show that a proper weight is essential for ECPE task.

\subsubsection{Runtime analysis}

To confirm that our method is more efficient than baselines, we perform runtime analysis among them. Note that our method needs only 5 train epochs on each fold, while baselines need 10 epochs. Therefore, we report the average running time on one epoch and total running time on all epochs on each fold. Results showed in Figure~\ref{subfig:right} suggests that our method reduce a 75\% and 55\% of time on total and average runtime on step two compared with previous methods \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}.

\begin{figure}[h]
\centering
\includegraphics[width=0.49\linewidth]{total_duration.png}
\includegraphics[width=0.49\linewidth]{average_duration.png}
\caption{Runtime on each fold of overall epochs and per epoch\label{subfig:right}}
\end{figure}

\subsubsection{Error analysis}

We are also curious about the reason of existing errors, therefore, we conduct further analysis on wrong pairs extracting in step two, traceback their origins, and classify errors into four categories. Table~\ref{error} shows detailed error analysis of two models, DQAN and Inter-EC\cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}. In this section, we mostly talk about False Negative samples (FN), which is also the major reason for error propagation as we talked in previous sections. Given a false negative sample $(c_e, c_c)$, there are four types of reasons, which can be divided into the fault of step one (3) and step two (1). The fault of step one represents the deficiency of parts of correct pairs sending into step two, including no emotion clause $(None, c_c)$ (no emo.), no cause clause $(c_e, None)$ (no cau.), neither emotion clause nor cause clause $(None, None)$ (no emo. and cau.). The fault of step two is wrong pairing that $f(c_e, c_c)=0$. We present the average number of errors per fold (avg.) in Table ~\ref{error}. The fault of step one account for 99.16\% of all errors in {\bf Inter-EC}, which proves that the first three types of errors are main issues of all wrong cases. Our DQAN model drops about 17.00\%, 12.45\% and 18.67\% on these three types of errors compared with {\bf Inter-EC} respectively, indicating that our model can alleviate these errors significantly. The decrease testifies the effectiveness of weighted loss to resist from error propagation.

\begin{table}[ht]
\centering
\caption{The average number and percentage of errors per fold.}
\label{error}
\begin{tabular}{c|c|c|c|c|c}
\hline
\diagbox [width=5em,trim=l] {Methods}{type} & no emo. & no cau. &  \tabincell{c}{no emo.\\ and cau.} & wrong pair & Total \\ \hline
\multirow{2}*{\bf Inter-EC} & 10.0 & 53.0 & 31.6 & 0.8 & 95.4 \\
& 10.48\% & 55.56\% & 33.12\% & 0.84\% & 100\% \\ \hline
\multirow{2}*{\bf DQAN} & 8.3 & 46.4 & 25.7 & 5.8 & 86.2 \\
& 9.63\%  & 53.83\% & 29.81\% & 6.73\% & 100\% \\ \hline
\end{tabular}
\end{table}

\subsection{Further results on ECE task}
\begin{table*}[h]
\centering
\caption{Experiments results on Chinese and English datasets on ECE task. Methods are divided into ECE based and ECPE based,  and the bold represents the highest performance on each type of methods. }\label{ECE}
\begin{tabular}{l|ccc|ccc} \hline
 & \multicolumn{3}{c|}{Chinese dataset} & \multicolumn{3}{c}{English dataset}  \\ \hline
Methods & P & R & F1 & P & R & F1 \\ \hline
 \emph{ECE methods}  & \multicolumn{6}{c}{} \\ \hline
 {\bf Word2vec \cite{word2vec}} & 0.4301 & 0.4233 & 0.4136 & 0.1651 & {\bf 0.8673} & 0.2774\\
{\bf SVM} & 0.4200 & 0.4375 & 0.4285 & 0.2757 & 0.6416 & 0.3856\\
{\bf CNN \cite{CNN}} & 0.6472 & 0.5493 & 0.5915 & 0.7218 & 0.2628 & 0.3390\\
{\bf Multi-Kernel \cite{SVM-Event-DrivenEmotionCauseExtractionwithCorpusConstruction}}& 0.6588 &0.6927 &0.6752 & - & - & -\\
{\bf Memnet \cite{AQuestionAnsweringApproachtoEmotionCauseExtraction}}& 0.5922 &0.6354 &0.6134 & - & - & -\\
{\bf ConvMS-M \cite{AQuestionAnsweringApproachtoEmotionCauseExtraction}} &0.7076 &0.6838 &0.6955 & 0.4605 &0.4177 & 0.4381\\
{\bf CANN \cite{ACo-AttentionNeuralNetworkModelforEmotionCauseAnalysiswithEmotionalContextAwareness}} &0.7721 &0.6891 &0.7266 & - & - & -\\
{\bf RTHN \cite{ARNN-TransformerHierarchicalNetworkforEmotionCauseExtraction}} &0.7697 & {\bf 0.7662} &0.7677 & - & - & -\\
{\bf MANN \cite{Context-AwareMulti-ViewAttentionNetworksforEmotionCauseExtraction}} & {\bf 0.7843} & 0.7587 & {\bf 0.7706} & {\bf 0.7933} & 0.4081 & {\bf 0.5328}\\
\hline
 \emph{ECPE methods} & \multicolumn{6}{c}{} \\ \hline
{\bf CANN-E \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}} &0.4826 &0.3160 &0.3797 & - & - & -\\
{\bf Inter-EC \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}}& 0.7041 &0.6083 &0.6507 & - & - & -\\ \hline
{\bf DQAN} &{\bf 0.7732} &{\bf 0.6370} & {\bf 0.6979}  &{\bf  0.8058} & {\bf 0.4324} & {\bf 0.5628}\\ \hline
\end{tabular}
\end{table*}

Since previous works conducted experiments on a Chinese ECE dataset\cite{SVM-Event-DrivenEmotionCauseExtractionwithCorpusConstruction} with their ECPE methods, we evaluate our model's performance on ECE tasks as well. Furthermore, we expand our experiments on another English dataset\cite{NTCIR}, which is also a benchmark dataset on ECE task. The details about the two datasets are summarized in Table \ref{ECE dataset}. The
metrics we used in evaluation follows\cite{Rule-AText-drivenRule-basedSystemforEmotionCauseDetection} with precision, recall and f1-score used as evaluation metrics.
\begin{equation}
\begin{aligned}
&P = \dfrac{\sum_{correct\_causes}}{\sum_{proposed\_causes}} \\
&R = \dfrac{\sum_{correct\_causes}}{\sum_{annotated\_causes}} \\
&F1 = \dfrac{2\times P \times R}{P+R} \\
\end{aligned}
\end{equation}
The metrics are similar to those in ECPE task. Proposed causes denotes the cause clauses predicted by the model, annotated causes denotes the cause clauses that are labeled in the dataset and correct causes means the clause that are both labeled and predicted as a cause clause.

\begin{table}[h]
\centering
\caption{Details about the two ECE datasets.}
\label{ECE dataset}
\begin{tabular}{l|c|c} \hline
 & Number & Percentage \\ \hline
 \emph{Chinese Dataset} & & \\ \hline
Documents & 2105 & 100\% \\
Documents with 1 cause clause & 2046 & 97.20\% \\
Documents with 2 cause clauses & 56& 2.61\% \\
Documents with 3 cause clauses & 3 & 0.14\% \\ \hline

\emph{English Dataset} & & \\ \hline
Documents & 2145 & 100\% \\
Documents with 1 cause clause & 1949 & 90.86\% \\
Documents with 2 cause clauses & 164& 7.65\% \\
Documents with 3 cause clauses & 32 & 1.49\% \\ \hline
\end{tabular}
\end{table}

We use pre-trained word2vec \cite{word2vec} for Chinese dataset and Glove \cite{Glove} for English dataset. Our method does not use any emotional annotations in the testing. The baselines are listed as follows:
\begin{itemize}
\item {\bf Word2vec} uses word representations obtained by Word2vec \cite{word2vec} as features and then trains an SVM classifier.
\item {\bf SVM} uses unigrams, bigrams, and trigrams as features and then trains an SVM classifier.
\item {\bf CNN} is the basic CNN proposed by \cite{CNN}. In this model, the candidate clause presentation and the emotion clause presentation are together as input.
\item {\bf Multi-kernel} uses the multi-kernel method to identify the cause  \cite{SVM-Event-DrivenEmotionCauseExtractionwithCorpusConstruction}.
\item {\bf Memnet} denotes a deep memory network \cite{AQuestionAnsweringApproachtoEmotionCauseExtraction}.
\item {\bf ConvMS-M} is a convolutional multiple-slot deep memory network \cite{AQuestionAnsweringApproachtoEmotionCauseExtraction}.
\item {\bf CANN}  denotes a co-attention neural network model \cite{ACo-AttentionNeuralNetworkModelforEmotionCauseAnalysiswithEmotionalContextAwareness}.
\item {\bf RTHN} is a joint framework, using RNN and Transformer to encode and classify multiple clauses synchronously \cite{ARNN-TransformerHierarchicalNetworkforEmotionCauseExtraction}.
\item {\bf MANN} employs a multi-attention-based model for emotion cause extraction\cite{Context-AwareMulti-ViewAttentionNetworksforEmotionCauseExtraction}.
\item {\bf CANN-E} \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts} is an extended model with a removal of annotated emotions from {\bf CANN}.
\item {\bf Inter-EC} is a hierarchical model mainly proposed for ECPE task  \cite{Emotion-CausePairExtraction:ANewTasktoEmotionAnalysisinTexts}.
\end{itemize}

The results are listed in Table~\ref{ECE}. We separate these methods into two types, ECE-based and ECPE-based, with the former used emotion signals and the latter not. In the English dataset, our approach achieve the best performance with 5.6\% improvement on f1-score and 1.6\% improvement on precision compared with the ECE-based baselines. It indicates that our ECPE-based method can overcome the deficiency of emotion annotations and even perform better on ECE task. In the Chinese dataset, {\bf DQAN} outperforms 7.3\% higher on f1-score compared with the best ECPE-based method and 84\% higher compared with {\bf CANN-E}, an ECE-based model with emotion annotation removed, which exhibits that by jointly adding semantic information and weighted loss, our method is also better to identify causes in such ECE tasks. Admittedly, the performance on the Chinese dataset is lower than a few ECE-based methods. One possible explanation is that such ECE methods may overfit dataset. From the details of the two datasets, the percentage of documents with more than one cause clause is only 2.75\% in the Chinese dataset while that number is 9.09\% in the English dataset, which indicates that the Chinese dataset is a simpler one compared with the English dataset. Moreover, the distance between single cause clause and emotion clause may be much close. Thus, such ECE approaches may perform well by overfitting the distance, while they perform badly when it comes to the English dataset with more complex and diverse cause clauses.

\section{Conclusions}\label{Conclusions}
In this paper, we propose a neural network named dual-questioning attention network for emotion-cause pair extraction task. Experiment results show that our method performs better than a range of baselines on ECPE task and even better on some evaluations than other methods on ECE task, even without essential emotion annotations. Further analysis demonstrates that a proper weight setting and contextual semantics are necessary.

\bibliographystyle{IEEEtran}