text:
Digital Archive Take It or Leave It: Institution, Image, Ideology
See all works by Jenny Holzer
Jenny Holzer's artistic medium is words and ideas, and she uses a variety of vehicles—from stickers, posters, and T-shirts to benches, bronze plaques, electronic displays, and the Internet—to disseminate her piercingly incisive phrases in public spaces. Rooted in conceptual art, semiotics, and feminism, Holzer's artistic practice interrogates the effects of rhetoric, engaging spectators in fundamental questions such as "Who is speaking?" "Where does this text come from?" and "What does it mean?" Answers, however, remain productively elusive. Her texts—whether directives, confessions, or observations—elide authorial intentionality and unlock a sort of societal subconscious, from which hacked-up bits of ideology, desire, fear, humor, and hatred pour forth.
Beginning with Truisms (1977–79), Holzer has authored dozens of textual series that—while distinct in personality, tone, and subject—often resurface in various material forms over time.# Her Inflammatory Essays (1979–81) are a series of mass-produced short essays—one hundred words in twenty lines—printed in capital letters on colorful paper and originally displayed anonymously in the streets.# The essays cover a variety of subjects that have been ongoing concerns throughout Holzer's career, including power, social control, abuse, consumption, and sex. Declarative and forceful in tone, they embody Holzer's distinctively crafted voice, one that is omniscient, detached, and yet enraged. "Don't talk down to me. Don't be polite to me. Don't try to make me feel nice. Don't relax. I'll cut the smile off your face," begins the essay on light pink paper. The "author" projected by these texts, however, is never clear but rather shifts between multiple, conflicting political and gender identities. The texts, as David Joselit has suggested, are "nowhere and everywhere—both uncannily personal and rigidly ideological."#
On left: Fred Wilson, Love and Loss in the Milky Way, 2005; on floor: Felix Gonzalez-Torres, "Untitled" (USA Today), 1990; right wall, from left to right: Jenny Holzer, Inflammatory Essays, 1979–81, and Laments: I am a man . . ., 1987. Installation view at the Hammer Museum, Los Angeles
Take It or Leave It: Institution, Image, Ideology. February 9–May 18, 2014. Photography by Brian Forrest
In Laments: I am a man . . . (1987), Holzer used the electronic LED signs for which she is best known to display her texts. Evocative of the information age's never-ending barrage of opinions, facts, and fictions, the proto-pixel technology adapted from commercial signage employs small diode lights to generate phrases that disappear as quickly as they arrive. Holzer has used technology to smuggle her urgent and provocative texts into unlikely sites, displaying, for instance, "Protect me from what I want" on the large electronic billboard in Times Square in 1986. In Laments: I am a man . . . , yellow text scrolls up an LED sign that is oriented vertically. The text, which repeats twice over the approximately six minutes of running time, begins "I am a man" and then continues with a series of short phrases, most of which begin with "I" and describe a selfishly motivated behavior or egocentric viewpoint: "I chase people around the house," "I can do it more than once," "I need perfection but when I implement it half of everyone dies," and, at the end, "I will kill you for what you might do." The work, first exhibited at Documenta 8, is part of the Laments series, a set of texts often displayed on both LED signs and stone sarcophagi that revolves around themes of sorrow, pain, power, and destruction.
In the current era of terror, Holzer has turned her attention to the erasure of language and information in declassified government documents. Part of her series of thirty-two Redaction Paintings, MEMORANDUM FOR CONDOLEEZZA RICE GREEN (2006) reproduces a magnified copy of a typed memo, dated January 25, 2001, from Richard C. Clarke of the National Security Council to then secretary of state Condoleezza Rice seeking clarification on terrorist threats. Screen-printed black text on bright green, the memo requests a review of the "al Qida network" (eight months before 9/11) and inquires about funding groups in opposition to "the Taliban/al Qida," such as the Afghan Northern Alliance, the perceived "threat magnitude" from the terrorist network, and the Bush administration's intended strategy. Holzer's Redaction Paintings—which include witness and soldier depositions, George W. Bush's opinion on the authority of the Geneva Conventions in a period of international terror, and even autopsy reports—index the same unsettling mix of violence, ignorance, and truth that has always intrigued Holzer and remained at the center of her work.
—Ruth Erickson
"Jenny Holzer." Take It or Leave It: Institution, Image, Ideology Digital Archive. Los Angeles: Hammer Museum, 2017. {PAGE_URL}.


text:
\section{Introduction}

Recently, fine-grained understanding of emotional clause level
~\citep{turcan-etal-2021-multi,poria2020beneath,poria2021recognizing}
has gained increasing attention from the research community,
especially a surge of research to determine
the cause of emotional expression in text
~\citep{bi2020ecsp,ding2020experimental,li2022neutral,uymaz2022vector}.

Emotion-Cause Pair Extraction (ECPE)
~\citep{xia-ding-2019-emotion}
is a new task that extracts all emotion clauses coupled with their
cause clauses from a given document.
For understanding simplicity, we following
~\citet{wei-etal-2020-effective,chen-etal-2020-conditional}
abstract the process of ECPE as four separate modules:
word-level encoding, clause-level encoding,
pair-level encoding, and pair extraction. The first three
parts separately model the word-to-word relationship,
clause-to-clause relationship, and pair-to-pair relationship.

\begin{figure}[hbp]
  \centerline{\includegraphics[width=0.5\linewidth]{task_definition.jpg}}
  \caption{A sample document with three sentences. Each sentence
  consists of two clauses. There are three types of
  ECPs inked red, purple, and green.}
  \label{figtask}
\end{figure}

For the pair-to-pair relationship, there have been numerous works
~\citep{fan-etal-2020-transition,ding-etal-2020-ecpe,fan2022combining,qie2022dcn,jin2022emotion}
to contribute particular advantages with their different networks.
However, for the clause-to-clause relationship,
most approaches use BiLSTM and only~\citet{wei-etal-2020-effective}
adopts GAT~\citep{velivckovic2017graph}.
From~\citet{thost2021directed,shen2021directed},
BiLSTM relatively neglects the remote clause, and GAT loses
the sequential information. As a result, clause-level encoding
is a fundamental yet under-explored part of the ECPE process.
And this imbalance is intelligible because pair-level encoding
has an exclusive direct influence on the pair extraction.

Moreover, from the findings of linguistics and psychologic:
~\citet{mann1988rhetorical,marcu2000theory}
have proposed that clauses or sentences are often indicative of an
underlying structure of one document;
~\citet{ruusuvuori2012emotion} has provided support that
the emotion of clauses tends to be constant in a sentence.
More recently, ~\citet{wu2022knowledge} has argued that the entire event
is usually across a set of clauses in a sentence.
~\citet{huang2022deep} has exploited general grammatical conventions
to span-encode sentences and clauses, demonstrating the existing
grammatical information contained in sentences and clauses.
Hence, advances in the clause-to-clause relationship,
specifically the knowledge of whether the clause
spans a sentence, are available for
distinguishing Emotion-Cause Pair (ECP) from the other non-ECPs.

To this end, we define a novel
clause-to-clause relationship, as shown in Figure\textcolor{red}{\ref{figtask}}.
Proceeding from the sentence attributions of two observed clauses,
we delimit clause-to-clause relationships as:
\begin{itemize}
  \item \textbf{Outer-clause relationship}: The two observed
  clauses are in different sentences.
  \item \textbf{Inter-clause relationship}: The two different observed
  clauses are in the same sentences.
  \item \textbf{Intra-clause relationship}: The two observed
  clauses are the same clause (self-influence).
\end{itemize}

Furthermore, \citet{chen2022recurrent,shi2022optimizing} argued
that the clause representations can be integrated with
the prediction sequence of two subtasks named emotion extraction
and cause extraction to achieve improvement.
Nevertheless, due to the fine-grained relationship representation
defined by ours
mismatching the binary-classification prediction sequence,
such the straightforward integrating hides the clues to emotions.
\citet{chen-etal-2020-unified} has reformulated binary classifiers
as 7-classifier and 4-classifier to fine the grain but
at the loss of generality.

Considering this problem,
and effectively empowering all approaches to utilize
our defined relationship's knowledge, we propose a clause-level
encoding model for ECPE, instead of contributing an end-to-end model.
Our model, composed
of \textbf{E}nhanced GAT and \textbf{A}ctivation Sort, is called EA-GAT,
which can be easily applicable to all existing approaches.
Enhanced GAT is a multi-mask GAT proposed to learn our new defined
clause-to-clause relationship. The Activation Sort is
a radical but valid pre-processing trick for achieving grain consistency.

Specifically, Enhanced GAT
yields a clause representation through separately
modeling the attention of 3 types of relationships.
And the Activation Sort is fed with emotion/cause prediction sequences
and outputs the auxiliary information for the second Enhanced GAT module.
It wides the gap among predictions via sort-based mapping, which
can explore different relationships of clauses beyond the polarity prediction.
Finally, EA-GAT produces a more favorable clause representation
for subsequent pair-level encoding.

To gauge the generality of our clause encoding model, we conduct
comprehensive experiments comparing with all published ECPE methods.
And we perform intrinsic contrast experiments of the adjacency matrix to
obtain insight into how EAGAT grasps the clause relationship.
Furthermore, we construct a new benchmark of the English dataset via
leveraging the RECCON proposed for Causal Emotion Entailment task.

Our main contributions can be summarized as follows:
\begin{itemize}
  \item To the best of our knowledge, we are first to
  define the clause-to-clause relationship and develop
  a multi-mask attention to learn their representation.
  \item We proposed the EA-GAT model, which can more effectively work
  in the clause-level encoding part of ECPE than networks adopted by previous methods.
  \item We conduct our comprehensive evaluation on the standard Chinese dataset
  and a newly introduced English dataset,
  which demonstrates that our method can be generally embedded into
  all existing approaches and significantly outperforms
  their original performance, including the state-of-the-art work.
\end{itemize}

The remainder of this paper is organized as follows:
in Section~\ref{rw}, we briefly introduce the related works;
in Section~\ref{td}, we formalize the task;
in Section~\ref{me}, we describe our proposed method;
in Section~\ref{ex}, we present the details and data of the experiments;
in Section~\ref{rd}, we analyze the results and limitations;
in Section~\ref{co}, we conclude of the proposed work.

\section{Related Works}\label{rw}

\begin{table*}
    \footnotesize
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{|l|c|c|c|c|}
      \hline
      \textbf{Model} & word-level encoding & clause-level encoding & pair-level encoding&clause extractions\\
      \hline
      Inter-EC \citep{xia-ding-2019-emotion} & word2vec       & BiLSTM & Logistic regression&\multirow{17}{*}{\thead{linear layer\\+softmax}} \\
      RankCP \citep{wei-etal-2020-effective}   & word2vec/BERT  & GAT              & Rank model&\\
      ECPE-2D \citep{ding-etal-2020-ecpe}  & word2vec/BERT  & BiLSTM & transformer&\\
      TDGC \citep{fan-etal-2020-transition}     & word2vec/BERT  & BiLSTM & Transition Model&\\
      IE-CNN \citep{chen-etal-2020-unified}   & word2vec       & BiLSTM & CNN&\\
      PairGCN \citep{chen-etal-2020-end}  & word2vec/BERT       & BiLSTM& GCN&\\
      SLSN \citep{cheng-etal-2020-symmetric}     & word2vec		  & BiLSTM & Local Searcher&\\
      JointNN \citep{tang2020joint}  & BERT    		  & BiLSTM+attention & Joint MLP Model&\\
      MLL \citep{ding-etal-2020-end} & word2vec       & BiLSTM & sliding window&\\
      CPAM \citep{chen-etal-2020-conditional}	 & word2vec       & BiLSTM & Aggreggation&\\
    RSN \citep{chen2022recurrent} & word2vec/BERT & BiLSTM &Linear&\\
    MASTM \citep{shi2022optimizing} & word2vec &BiLSTM &3-level filters&\\
    PTF \citep{wu2023pairwise} & word2vec/BERT & BiLSTM & PTF-based model & \\
    CL-ECPE \citep{zhang2022cl} & word2vec & BiLSTM & adversarial model &\\
    BERT+ \citep{cao2022research} & BERT & BiLSTM & MOO &\\
    MGSAG \citep{bao2022multi}&word2vec/BERT & BiLSTM& Semantic Aware Graph &\\
    SAP-ECPE \citep{huang2022deep}&word2vec&BiLSTM&BiGRU&\\
    \hline
    \end{tabular}}
    \caption{ 4 parts of existing approaches. There are different networks for encoding the pair-to-pair representations whereas clause-level encoding sharply lacks effective research. }
    \label{tabrelated}
  \end{table*}

The process of ECPE task has been carried out in $3$ encoding parts
and $1$ extraction part, as shown in Table\ref{tabrelated}.
In word-level encoding, word representation adopts
pre-training with word2vec \citep{NIPS2013_9aa42b31} toolkit or
$BERT$~\citep{devlin-etal-2019-bert}.
Moreover, almost all approaches adopt BiLSTM to model the relationship
between clauses in clause-level encoding.
But pair-level encoding has attracted
substantial interest from numerous researchers, in which
\citet{fan-etal-2020-transition} has transformed ECPE into a procedure of directed graph structure and adopted a transition-based model to identify the semantic relationship between emotions and causes;
\citet{wei-etal-2020-effective} has used the RankNN with an RBF kernel function for the pair-level encoding. Specifically, they proposed a GAT to capture the underlying relationship among different clauses via designing a full-connected graph;
\citet{chen-etal-2020-end} has adopted GNN framing pairs as nodes to learn the pair-level representation.

And to alleviate the loss of information by binary classification,
\citet{chen-etal-2020-unified} used a more fine-grained tagging scheme that combines emotion labels and cause labels with emotion types separately, and
\citet{chen-etal-2020-conditional} has defined a new task of determining whether or not an ECP has a valid causal relationship under different contexts.
\citet{wu2023pairwise} propose a labeling framework tackling the whole
ECPE in one unified tagging task to optimize the ECPE task globally and extract more accurate ECPs.
Based on these attempts, increasing works have been made to
design the extra annotation~\citep{chang2022emotion,huang2022deep} or auxiliary dataset~\citep{cao2022research,wu2022knowledge,zhang2022cl} or keywords~\citep{bao2022multi}.

More recently, the clause-level relationship between emotions and causes has gained
significant attention due to its effectiveness in real-world applications.
~\citet{kumar2022emotion} has replaced the traditional technique
for rule-based semantic analysis work on sentence-level
with the ECPE-BERT model to recognize emotion in psychological texts.
And~\citet{ghosh2022cares} has proposed a pre-trained transformer-based
ECPE model to address the problem of cause annotation and cause extraction for emotion in suicide notes datasets. Furthermore, inspired
by recent advances in ECPE task,~\citet{halat2022multi} has analyzed the
causes leading to a negative statement via ECPE model, to improve the
recognition of hate speech and offensive language (HOF).

Of 4 parts of the ECPE process, our model, as a clause-level encoding method, is broadly applicable
to all existing works well, reserving
the crucial components of each work
and exploiting the clause-to-clause relationships.

\section{Task Definition}\label{td}

Given a document $D=(c_{1},c_{2},\dots,c_{|D\vert })$
where $D$ is the number of
clauses and the $i$-th clause
$c_{i}=(\omega^{i}_{1},\omega^{i}_{2},\dots,\omega^{i}_{|c_{i}\vert })$
is a word sequence,
the goal of ECPE is to extract
a set of emotion-cause pairs (ECPs) in $D$:

\begin{equation}
  P=\left\{\dots,(c_{i}^{e},c_{j}^{c}),\dots \right\}
  \label{eqn1}
\end{equation}

where $(c_{i}^{e},c_{j}^{c})$ is an ECP consisting of
$i$-th clause $c_{i}^{e}\in D$ as emotion clause
and $j$-th clause $c_{j}^{c}\in D$ as corresponding cause clause.
Additionally, the input of our method is
$(c_{1},c_{2},\dots,c_{|D\vert })$ outputting
from word-level context encoding,
and the output of our method is clause representation
(${h^{'}_{c_{1}},h^{'}_{c_{2}},\dots,h^{'}_{c_{|D|}}}$).
Note that the pair representation $P_{i,j}$, the input
of pair-to-pair encoding, is contacted by
clause representation $h^{'}_{c_{i}}$ and $h^{'}_{c_{j}}$,
which is entirely different from clause representation and
not developed in this paper.

\section{Methodology}\label{me}

\begin{figure*}[hbp]
    \includegraphics[width=0.95\linewidth]{overall_structure.jpg}
    \caption{Overview of our model with the orange-dash box
    indicates our model, and the gray box represents
    our model-irrelevant parts in ECPE processing.
    The input of our model is individual clause
    embedding integrated by word-level encoding and the output of
    our model is the optimal clause representation for
    pair-level encoding. }
    \label{figoverall}
  \end{figure*}

The main framework of EA-GAT consists of hierarchical \textbf{E}nhanced
GAT modules and \textbf{A}ctivation Sort modules.
The overview is shown in Figure\ref{figoverall}. Enhanced GAT is used to
process clauses under the clause-to-clause relationship
defined in Section\ref{sec4.1} and Activation Sort enlarges the
spread of predictions weighting of negative samples to form the
new adjacency matrix for Enhanced GAT.
Furthermore, an additional sort loss
is adopted to evaluate the performance of the whole framework.

\subsection{Building Clause-to-Clause Relationship}\label{sec4.1}

In the tokenizer, 3 sentence-ending dot marks, ``$.$'',``$?$'' and``$!$'',
are denoted by a token <$period$>.
For notational consistency, we also add <$period$> to the beginning
and the end of a document.
Hence, for any sentence $S$, one can partly define a unique sequence order
on a document $D=(<period>,c_{1},c_{2},\ldots,<period>,c_{i},\ldots,c_{j},<period>,\ldots,c_{|D|},<period>)$,
such that all clauses between two contiguous <$period$> belong to
the same $S$.
Then, a document
can also be denoted by $D=(S_{1},S_{2},……,S_{n})$,
where $n$ is one less the number of <$period$>.

We formalize the clause-to-clause relationship by the
multi-mask matrix
$M=\in \mathbb{R}^{|D|\times |D|}$ as follows:

\begin{equation}
  \begin{split}
    M_{i,j}=\left\{
      \begin{array}{lr}
        2, &S_{i}\neq S_{j}\\
        1, &S_{i}=S_{j}  \bigwedge  c_{i}\neq c_{j}\\
        0, &c_{i}=c_{j}
      \end{array}
    \right.
  \end{split}
  \label{eqn2}
\end{equation}

where $M_{i,j}$ denotes the relationship type from $c_{j}$ to $c_{i}$.
There are 3 diverse values $(2,1,0)$ that separately represent
outer-clause relationship, inter-clause relationship,
and intra-clause relationship.

\subsection{ Enhanced Graph Attention}

Given a clause embedding $D=(c_{1},c_{2},\dots,c_{|D\vert })$
generated by word-level encoding,
we design Enhanced GAT to instantiate the
information aggregation of clauses.

Specifically, for each clause, self-attention  aggregates
information from neighboring clauses to learn the
updated attention weight as follows:

\begin{equation}
  A^{(0)}_{ij}=\frac{LeakyReLU(e_{i,j}^{(0)})}{\sum_{k\in|D|}LeakyReLU(e_{i,k}^{(0)})}
\label{eqn3}
\end{equation}

\begin{equation}
  e_{i,j}^{(0)}=W^{(0)}_{i(row)}\overrightarrow{c}_{i}+W^{(0)}_{j(col)}\overrightarrow{c}_{j}
  \label{eqn4}
\end{equation}

Where $A^{(0)}$represents the adjacency matrix modeled
by graph attention, the superscript $(0)$ denotes
the first E-GAT module ($(1)$ denotes the second),
and $A^{(0)}_{ij}$ measures
the importance of influence of the $j$-th clause
on the $i$-th clause,
$W^{(0)}_{row}$ and $W^{(0)}_{col}$ are the
learnable parameters in the graph attention.

Considering the multi-mask $M$, the primary clause representation
$h$ is the sum of 3 types of relationship attention operators:

\begin{equation}
  h=\sum_{m=0,1,2} eLU(A_{m}^{(0)}DW^{(0)}_{m})
  \label{eqn5}
\end{equation}

\begin{equation}
  \begin{split}
    A^{(0)}_{m(i,j)}=\left\{
      \begin{array}{lr}
        A^{(0)}_{i,j}, &M_{i,j}=m\\
        0, &M_{i,j}\neq m
      \end{array}
    \right.
  \end{split}
  \label{eqn6}
\end{equation}

Where the  $W^{(0)}_{m}$ denotes the learnable parameters for
each mask value $m$. Following the strategy of other ECPE approaches
which performs two individual subtasks named emotion/cause extractions,
the probabilities of each clause for the two subtasks are computed as follows:

\begin{equation}
  \widehat{y}^{e}=Sigmoid(W_{e}h+b_{e})
  \label{eqn7}
\end{equation}

\begin{equation}
  \widehat{y}^{c}=Sigmoid(W_{c}h+b_{c})
  \label{eqn8}
\end{equation}

Where $\widehat{y}^{e}$ and $\widehat{y}^{c}$ are
prediction sequences of emotion/cause extraction,
$W_{e}$, $W_{c}$, $b_{e}$ and $b_{c}$ are learnable parameters
of two linear layers.

\subsection{Activation Sort}\label{subas}

We use prediction sequences
$\widehat{y}^{e}$ and $\widehat{y}^{c}$
to generate an enhanced adjacency matrix.
Because as an individual subtask,
$\widehat{y}^{e}_{i}$
denotes the probability that the $i$-th clause
is an emotion clause. Similarly,
$\widehat{y}^{c}_{j}$
denotes the probability that the $j$-th clause
is the cause of a certain emotion clause.
For an ECP $(c_{i}^{e},c_{j}^{c})$,
it should more possibly consist of the $c_{i}^{e}$
with a higher $\widehat{y}^{e}_{i}$ and the $c_{j}^{c}$
with a higher $\widehat{y}^{c}_{j}$.

But the ground truth sequences $y^{e}$ and $y^{c}$ are binary,
~\citet{chen-etal-2020-unified,chen-etal-2020-conditional} have
provided the supports of which
framing emotion/cause extractions as binary classification
partly ignores the relationship from cause to a specific emotion.

In order to make emotion/cause predictions more efficient
to indicate the relationship among all clauses, especially non-ECPs.
We design a trick processing the emotion/cause predictions to provide
EGAT auxiliary information as part of input.
In the output of each EGAT module, We sort all predictions
in $\widehat{y}^{e}$ and $\widehat{y}^{c}$ to expand the
differences, which contributes to the next EGAT to generate more favorable adjacency.
Our sort approach named $Activation Sort$ is
operated as follows:

\begin{equation}
  \begin{aligned}
  \widehat{r}^{e}_{i}=label[argsort(\widehat{y}^{e})]
  \end{aligned}
  \label{eqn9}
\end{equation}

\begin{equation}
  \begin{aligned}
  \widehat{r}^{c}_{j}=label[argsort(\widehat{y}^{c})]
  \end{aligned}
  \label{eqn10}
\end{equation}

\begin{equation}
  \begin{aligned}
  label=[1,2,3,\dots,|D|]
  \end{aligned}
  \label{eqn11}
\end{equation}

Specifically,
we decreasingly sort the $\widehat{y}^{e}$ and $\widehat{y}^{c}$,
and separately give each
$\widehat{y}^{e}_{i}$ and $\widehat{y}^{c}_{j}$
a serial number $\in [1,|D|]$
according to the sorting position,
use this serial number as an index,
and make the value of the corresponding index in the $label$
assigned to $\widehat{y}^{e}$ and $\widehat{y}^{c}$.
We design the $label$ as a sequence of natural numbers
not containing $0$,
which can distinguish the difference between each prediction.
A sort position embedding is adopted
for $\widehat{r}^{e}$ and $\widehat{r}^{c}$
to regulate magnitude order.
And after experimental trying, We also defined
an alternative $label$ sequence as follows:

\begin{equation}
  \begin{aligned}
  label[i]=2^{i}(i=[\log^{|D|}_{2}]-[\log^{|D|-i+1}_{2}])
  \end{aligned}
  \label{eqn12}
\end{equation}

We exemplify this equation by a document with $10$ clauses,
the value of $label$ is $[1,1,1,2,2,2,2,4,4,8]$.
In this way,  $\widehat{r}^{e}$ and $\widehat{r}^{c}$
have more visible differences compared with
the original prediction sequence close to $[0,0,0,0,0,0,0,0,1,1]$ (two positive example clauses).

Note that: (i) there are countless  $label$ sequences
that can expand the difference not be put forward;
(ii) compared with popular activation functions
such as $sigmoid$ or $tanh$, $Activation Sort$ can not
only expand the distance among non-ECPs after normalization
but also narrow down the distance between ECPs and non-ECPs;
(iii) the gradient stopping caused by argsort is a benefit
to eliminate the negative impact of
$Loss_{non-ECP}$ on $P^{0}_{y^{pair}}$
(detail shown in Section\ref{sec4.5});
(iv) in the pre-output of subtasks, $\widehat{y}^{e}$ and $\widehat{y}^{c}$
reserve the original prediction sequences.

\subsection{Enhanced GAT with Activation Sort}

After sorting $\widehat{y}^{e}$ and $\widehat{y}^{c}$,
activated clause-level relationship
(adjacency) matrix $e^{(1)}_{i,j}$is
denoted as:

\begin{equation}
  e^{(1)}_{i,j}=W^{(1)}_{i(row)}\widehat{r}^{e}_{i}+W^{(1)}_{j(col)}\widehat{r}^{c}_{j}
  \label{eqn13}
\end{equation}

\begin{equation}
  A^{(1)}_{ij}=\frac{LeakyReLU(e_{i,j}^{(1)})}{\sum_{k\in|D|}LeakyReLU(e_{i,k}^{(1)})} +A^{(0)}_{i,j}
\label{eqn14}
\end{equation}

Where $W^{(1)}_{i(row)}$ and $W^{(1)}_{j(col)}$
are learnable parameters that separately represent
the correlation of clauses in $\widehat{r}^{e}_{i}$
and $\widehat{r}^{c}_{j}$.
Under a hypothesis that $\widehat{y}^{e}$ and $\widehat{y}^{c}$
are close to ground truth sequences $y^{e}$ and $y^{c}$ (from
the empirical result of Section\ref{sec5.3}, it is a relatively weak
hypothesis),  $A^{(1)}$ depicts a learnable clause-to-clause
relationship adjusted by the emotion/cause ground truth.

Then we feed $A^{(1)}$ into the Enhanced GAT again, aiming to
generate the clause representation $h^{(1)}$ as follows:

\begin{equation}
  h^{(1)}=\sum_{m=0,1,2} eLU(A_{m}^{(1)}DW^{(1)}_{m})
  \label{eqn15}
\end{equation}

where $h^{(1)}$ is the output as enhanced clause representation
that can be used in the follow-up pair-level encoding
and extraction. The empirical case in Section\ref{sec5.5}
holds the interpretability of $h^{(1)}$ and $h$. On the whole,
Starting with the $2$-th EGAT, each adjacency matrix will be added the
auxiliary information from the output of the previous EGAT. We denote
the layers of EGAT as $l=2,3,\dots,L$, the clause representation of
each EGAT can be writen as:

\begin{equation}
  h^{(l)}=\sum_{m=0,1,2} eLU(A_{m}^{(l)}DW^{(l)}_{m})
  \label{eqn15}
\end{equation}

\begin{equation}
  A^{(l)}_{ij}=\frac{LeakyReLU(e_{i,j}^{(l)})}{\sum_{k\in|D|}LeakyReLU(e_{i,k}^{(l)})} +A^{(l-1)}_{i,j}
\label{eqn14}
\end{equation}

\subsection{Optimization}\label{sec4.5}

\begin{figure*}[hbp]
    \includegraphics[width=\linewidth]{sort.jpg}
    \caption{The forward processing and back propagation processing
    of our model (We only show the first and second layers of EGAT). }
    \label{figsort}
  \end{figure*}

Generally, ECPE task is optimized by 3 cross-entropy errors:
$Loss_{e}$, $Loss_{c}$ and $Loss_{pair}$,
where $Loss_{e}$ and $Loss_{c}$ measure
 $\widehat{y}^{e}$ and  $\widehat{y}^{c}$.  $Loss_{pair}$
measures the output of pair prediction sequence.

\begin{equation}
  Loss_{e}=\sum_{l = 1}^{L}\sum_{i \in |D|} -(P^{l}_{y^{e}_{i}} \log(1-P^{l}_{\widehat{y}^{e}_{i}})+(1-P^{l}_{y^{e}_{i}}) \log P^{l}_{\widehat{y}^{e}_{i}}  )
  \label{eqn16}
\end{equation}

\begin{equation}
  Loss_{c}=\sum_{l = 1}^{L}\sum_{i \in |D|} -(P^{l}_{y^{c}_{i}} \log(1-P^{l}_{\widehat{y}^{c}_{i}})+(1-P^{l}_{y^{c}_{i}}) \log P^{l}_{\widehat{y}^{c}_{i}}  )
  \label{eqn16}
\end{equation}

\begin{equation}
  Loss_{pair}=\sum_{l = 1}^{L}\sum_{i,j \in |D|\times|D|} -(P^{l}_{y^{pair}_{i,j}} \log(1-P^{l}_{\widehat{y}^{pair}_{i,j}})+(1-P^{l}_{y^{pair}_{i,j}}) \log P^{l}_{\widehat{y}^{pair}_{i,j}}  )
  \label{eqn16}
\end{equation}

In addition,
we propose sort loss measure the performance that
the $h^{'}$ exceeds $h$ as follows:

\begin{equation}
  Loss_{sort}=\sum_{l = 2}^{L} max(0,P^{l-1}_{y^{pair}_{i,j}=1}-P^{l}_{y^{pair}_{i,j}=1})+0.05
  \label{eqn16}
\end{equation}

where $P_{y^{pair}_{i,j}=1}$ is the prediction
of the labeled clause pair (ECP), $P^{l-1}$ is the output
from $h^{l-1}$, and $P^{l}$ is the output
from $h^{l}$.
$Loss_{sort}$ ensures that $P^{l}$ is more effective
than $P^{l-1}$,
and ultimately ensures that $A^{(l)}$ is more effective
than $A^{(l-1)}$.

Note that $Activation Sort$ has no gradient as shown in Figure~\ref{figsort},
so $Loss_{sort}$ only measures the parameters after
$Activation Sort$ in forward, and the parameters
of the previous Enhanced GAT module not be affected.
Due to the characteristics of $Loss_{sort}$,
which increases $P^{l}_{y^{pair}}$
and also inhibits the improvement of $P^{l-1}_{y^{pair}}$.
It is gradient stopping
that can prevent the deviate gradient descent
of the $Loss_{sort}$ from conflicting
with the other three losses.

\section{Experiments}\label{ex}

Extensive experiments are conducted to verify the
generality and effectiveness of our model
on the Chinese ECPE dataset proposed by~\citet{xia-ding-2019-emotion}
and the English RECCON dataset (ECPE-like task)
proposed by~\citet{poria2021recognizing}.
And we use the same metrics named $F1$ score for evaluating Chinese corpus following \citet{xia-ding-2019-emotion}
and $avgF1$ score (the average of $F1$ scores for both positive and negative examples) for evaluating English corpus following \citet{poria2021recognizing}.
\subsection{Dataset}

\begin{table}[hbp]
    \footnotesize
    \centering
    \begin{tabular}{|c|c|c|c|c|}
      \hline
      \textbf{dataset}&\textbf{document} & \textbf{sentence} & \textbf{clause} &\textbf{ECP}\\
      \hline
      Chinese&1945&8075&28727&2156\\
      English&1106&5513&11104&5380\\
      \hline
    \end{tabular}
    \caption{ Statistics of two datasets, where ECP stands for the
    emotion-cause pair.}
    \label{tab2}
  \end{table}

  \begin{figure}[hbp]
    \centerline{\includegraphics[width=0.5\linewidth]{statistics.png}}
    \caption{Frequency diagram of sampling in contiguous 10
    index of documents, where horizontal axis denotes the rate of
    sample numbers of 10 documents group, vertical axis stands for the
    correspond frequency.}
    \label{figfre}
  \end{figure}

  \begin{table*}[hbp]
    \footnotesize
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
      \hline
      \textbf{Statistic}&\textbf{fold1}&\textbf{fold2}&\textbf{fold3}&\textbf{fold4}&\textbf{fold5}&\textbf{fold6}&\textbf{fold7}&\textbf{fold8}&\textbf{fold9}&\textbf{fold10}\\
      \hline
      mean&342.78& 323.11& 353.30& 338.88& 340.09& 369.71& 326.13& 321.85& 336.10&338.4\\
      std&257.62& 246.80& 257.40& 251.27& 258.12& 246.92& 248.42& 239.52& 236.03&239.5\\
      skew ($\times10^{4}$)&1.71&2.17&1.66&2.39&2.07&1.14&2.45&2.21&2.04&2.11\\
      kurt ($\times10^{5}$)&-1.49&-1.49&-1.49&-1.49&-1.49&-1.49&-1.49&-1.49&-1.49&-1.49\\
      \hline
      std (Figure.1)&0.086& 0.086& 0.084& 0.077& 0.082& 0.091& 0.080& 0.083& 0.086&0.086\\
      \hline
  \end{tabular}}
  \caption{ Statistic in random 10-fold, we summarize the mean value,
  standard deviation, skewness and kurtosis of the test dataset document-ids. Additionally,
  we add the standard deviation of the sample frequencty. }
  \label{tab10fold}
  \end{table*}

  ~\citet{xia-ding-2019-emotion} constructs the Chinese dataset
based on the ECE corpus~\citep{gui-etal-2016-event} and
publishes the word segmentation version without periods. But
to help reproduction and discussions, we refer to the
ECE corpus and complete the ``.'', ``?'' and ``!''
to state the end of a sentence.

While the Chinese ECPE corpus is a well-known and pioneering dataset
for the ECPE task, it increasingly reveals some limitations that
complicate the current works. Sparsity is the most concerning
point. Statistically, in ECPE corpus, a document involves 14.32 clauses,
3.56 sentences but 1.11 labeled pairs. And following the
original splits, id-continuous samples are grouped together, when
many long and complete documents are divided into 2-5 id-continuous samples.
Hence, the sample variance is too numerically large and the
feature distribution is substantially different between the
train split and test split. Under the random initialization,
the trivial solution exists, leading to an unstable learning process.
Some works conditionally avoid this by deleting several samples from
train split, while we adopt an alternative strategy that more makes sense.
We initial the learnable parameters of our model with a pre-trained
model from few-shot learning with 800 samples in train split.

Besides, the English
dataset is about dyadic conversations. So we
define all utterances from the same speaker
belong to a sentence. The details of two datasets
are shown in Table\ref{tab2}. It has the
almost 17 times less ratio of non-ECPs to ECPs than ECPE datasets.
So although it is a neonate in the exploitation, it is more potential
for relationship discovery and construction. But the original split
is 3-fold, and the test dataset uses a contiguous sample strategy
making us concerned about whether the bias exists to noise the experiment.
To this end, we conduct a random 10-fold split for our and subsequent
research. Table.\ref{tab10fold} demonstrates that our split is stochastic
in the document distribution and provides the crucial statistics.
Figure,\ref{figfre} visualizes the sampling frequency in each 10-contiguous
samples, which indicates that our sampling is also independent.

In order to obtain statistically
credible performance, we evaluate our approach 10 times with
different data splits by separately following
~\citet{xia-ding-2019-emotion,poria2021recognizing}. Additionally,
we perform one sample $t-$test with $p$<0.01 to ensure the
significance.

\subsection{Implementation Details}

We replace the clause-level encoding part of
all existing approaches with our model,
with reservation of the other strategies and hyperparameters
of corresponding existing approaches, besides
the dimensions of E-GAT set to 768.
For instance, when we embed our model into RankCP, we keep the layer of GAT with 2,
learning rate with 0.001, weight decay with 1e-5, and the RBF kernel with $\sigma_{K}=1$.
As the same as the word embedding of corresponding dataset baseline,
we use $BERT_{Chinese}$ pre-trained model
\footnote{\url{http://github.com/huggingface/pytorch-pretrained-BERT}},
word vectors pre-trained from Chinese Weibo
\footnote{\url{http://www.aihuang.org/p/challenge.html}} for the experiment without BERT
in the Chinese dataset, and $RoBERTa_{base}$ pre-trained model
\footnote{\url{https://huggingface.co/roberta-base}}
in the English dataset.

And our entire codes and dataset splits are shared
\footnote{\url{https://drive.google.com/file/d/1S_l59YYZfOXiS1HAFz39Y3g9GdqtCqGI/view?usp=sharing}}.
Because the initial checkpoint models take up too much storage space. We will
publish them to Github after the final version.

\subsection{Baselines}

We compare our model with almost all published works, which can be
divided into two categories: one is standard baselines following the
benchmark~\citet{xia-ding-2019-emotion}, the other one is designed some
auxiliary annotation or proposed extra data samples. We detailed them
as follows:

\textbf{Inter-EC}. A two-step method proposed by~\citet{xia-ding-2019-emotion}
is the first to extract emotions and causes separately. And then to form
an ECP, it train a binary classifier to filter out ECPs and non-ECPs.

\textbf{RankCP}. A GAT-based method~\citep{wei-etal-2020-effective}
tackles ECP prediction from a ranking perspective, adopting a
graph attention network to propagate information among clauses.

\textbf{ECPE-2D}. A transformer-based method~\citep{ding-etal-2020-ecpe} to
construct a pairs matrix and achieve the interaction between ECPs by
leveraging a 2D transformer module.

\textbf{TDGC}. A transition-based method~\citep{fan-etal-2020-transition}
formulated the ECPE task as a set of actions and transitions with directed
graph construction.

\textbf{PairGCN}. A GCN-based method~\citep{chen-etal-2020-end} constructed
the influence relationships among candidate pairs by regarding pairs
as nodes.

\textbf{SLSN}. A local search method~\citep{cheng-etal-2020-symmetric}
extracted the local ECPs by two novel corss-subnetwork of symmetric
subnetworks separately.

\textbf{SAP}. A relative position method~\citep{huang2022deep} extracted
ECPs by defining the spans of two clauses among all pairs to achieve
better performance.

\textbf{RSN}. A recurrent-based method~\citep{chen2022recurrent} performs
multiple rounds of inference to recognize emotion clauses, cause clauses,
and emotion-cause pairs iteratively.

\textbf{JointNN}. A self-attention-based method~\citep{tang2020joint}
adopted BERT as word-level encoder and proposed a joint attention
model for pair-level encoding.

\textbf{MLL}. A slide window method~\citep{ding-etal-2020-end} employs
an emotion-pivoted cause extraction framework and a cause-pivoted emotion
extraction framework.

\textbf{CPAM}. A new benchmark~\citep{chen-etal-2020-conditional} determined
whether or not an ECP has a causal relationship given some specific
context clauses in a corresponding improved dataset via manual annotation.

\textbf{IE-CNN}. A new benchmark~\citep{chen-etal-2020-unified} proposed
a new tagging strategy including four causal identity labels and seven
emotion type labels.

\textbf{BERT+}. A BERT-based model~\citep{cao2022research} achieved
better performance following the benchmark as CPAM.

\textbf{CL-ECPE}. A new contrastive benchmark~\citep{zhang2022cl}
used adversarial samples as new datasets to perform the pair extraction.

\textbf{MGSAG}. A new benchmark~\citep{bao2022multi} incorporated
fine-grained and coarse-grained semantic representation by extracting
keywords.

\subsection{Overall Results}\label{sec5.3}

\begin{table*}[h]
    \footnotesize
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{|c|c|ccc|ccc|ccc|}
      \hline
      \multirow{2}{*}{\textbf{Category}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c}{emotion extraction(\%)} & \multicolumn{3}{c}{cause extraction(\%)} & \multicolumn{3}{c}{pair extraction(\%)}\\
      \cline{3-11}
      & & P & R  & F1 &P  & R  & F1 &P  & R  & F1\\
      \hline
      \multirow{16}{*}{\thead{without\\BERT}} & Inter-EC & 82.63 & 81.32 & 81.92 & 69.24 & 61.04 & 64.75 & 66.11 & 56.22 & 60.64\\
                             &+Ours & \textbf{82.89} & \textbf{81.94} & \textbf{82.41$\pm$1.75} & \textbf{71.33} & \textbf{65.07} &\textbf{68.06$\pm$2.41} & \textbf{68.47} & \textbf{59.31} & \textbf{63.56$\pm$2.11}\\
  \cline{2-11}
                                    & RankCP        & 87.03 & 84.06 & 85.48 & 69.27 & 67.43 & 68.24 & 66.98 & 65.46 & 66.10\\
                            & +Ours   & \textbf{88.16} & \textbf{85.87} &\textbf{87.00$\pm$1.65} &\textbf{72.82} & \textcolor{red}{\textbf{74.35}} &\textcolor{red}{\textbf{73.58$\pm$2.94}} &\textbf{69.54} & \textcolor{red}{\textbf{68.91}} & \textbf{69.22$\pm$2.53}\\
  \cline{2-11}
                                    & ECPE-2D  & 85.12 &\textbf{82.20} & 83.58 & \textbf{72.72} & \textbf{62.98} & \textbf{67.38} & 69.60 & \textbf{61.18} & 64.96\\
                            & +Ours  & \textbf{86.33} & 81.15 & \textbf{83.66$\pm$1.14} & 72.69 & 62.54 & 67.23$\pm$1.58 & \textbf{70.36} & 61.05 & \textbf{65.38$\pm$1.67}\\
  \cline{2-11}
                                    & TDGC     & 80.80 & 84.39 & 82.56 & 67.42 & 65.34 & 66.36 & 65.15 & 63.54 & 64.34\\
                            & +Ours     & \textbf{81.55} & \textbf{84.93} &\textbf{83.21$\pm$1.73} & \textbf{68.91} & \textbf{69.55} & \textbf{69.22$\pm$2.12} & \textbf{68.02} & \textbf{65.21} & \textbf{66.59$\pm$2.09}\\
  \cline{2-11}
                                    & PairGCN  & 85.87 & 72.08 & 78.29 & 72.83 & 59.53 & 65.41 & 69.99 & 57.79 & 63.21\\
                            & +Ours  & \textbf{86.39} & \textbf{73.31} & \textbf{79.31$\pm$1.66} & \textbf{76.41} & \textbf{64.28} & \textbf{69.82$\pm$2.27} & \textbf{71.06} & \textbf{62.41} & \textbf{66.45$\pm$1.95}\\
  \cline{2-11}
                                    & SLSN     & 84.06 & 79.80 & 81.81 & 69.92 & \textbf{65.88} & 67.78 & 68.36 & 62.91 & 65.45\\
                            & +Ours    & \textbf{84.74} & \textbf{80.69} & \textbf{82.67$\pm$1.33} & \textbf{73.53} & 64.21 &\textbf{68.55$\pm$2.03} & \textbf{70.00} & \textbf{63.68} & \textbf{66.69$\pm$1.79}\\
  \cline{2-11}
                                    & SAP   & \textbf{86.31} & \textbf{81.58} & \textbf{83.83} & 70.11 & \textbf{64.42} & 67.09 & \textbf{72.18} & 58.92 & 64.75\\
                                    & +Ours   & 85.59 & 79.12 & 82.23$\pm$1.63 & \textbf{72.39} & 64.29 & \textbf{68.10$\pm$2.59} & 71.56 & \textbf{62.93} & \textbf{66.97$\pm$2.31}\\
  \cline{2-11}
                                    &RSN       &86.88  & 87.43& 87.07& 73.62 & 65.54& 69.26& \textcolor{red}{72.15}& 63.77& 67.62\\
                &+Ours       & \textcolor{red}{\textbf{89.72}} & \textcolor{red}{\textbf{88.95}} & \textcolor{red}{\textbf{89.33$\pm$2.13}} & \textcolor{red}{\textbf{73.86}} & \textbf{69.46} & \textbf{71.59$\pm$2.56} & \textbf{71.55} & \textbf{67.42} & \textcolor{red}{\textbf{69.42$\pm$2.25}} \\
  \cline{2-11}
      \hline
      \multirow{14}{*}{\thead{with\\BERT}}    & JointNN  & \textbf{89.90} & 80.00 & 84.70 & $-$   & $-$   & $-$   & 71.10 & 60.70 & 65.50\\
                             & +Ours  & 88.25 & \textbf{84.56} & \textbf{86.37$\pm$2.51} & $-$   & $-$   & $-$   & \textbf{74.58} & \textbf{66.21} &\textbf{70.15$\pm$3.06}\\
  \cline{2-11}
                            & RankCP        & 91.23          & 89.99 & 90.57 & 74.61 & 77.83 & 76.15 & 71.19 & 76.30 & 73.60\\
                            & +Ours   & \textcolor{red}{\textbf{92.70}} & \textbf{91.67} &\textcolor{red}{\textbf{92.16$\pm$2.82}} &\textbf{77.39} & \textcolor{red}{\textbf{81.12}} &\textcolor{red}{\textbf{79.18$\pm$1.45}} &\textbf{74.25} & \textcolor{red}{\textbf{79.79}} & \textcolor{red}{\textbf{76.88$\pm$1.46}}\\
  \cline{2-11}
                                    & ECPE-2D  & 86.27 & \textcolor{red}{\textbf{92.21}} & \textbf{89.10} & 73.36 & 69.34 & 71.23 & 72.92 & 65.44 & 68.89\\
                            & +Ours  & \textbf{88.59} & 89.12 & 88.85$\pm$3.59 & \textbf{74.55} & \textbf{69.82} & \textbf{72.11$\pm$2.01} & \textbf{73.62} & \textbf{66.08} & \textbf{69.65$\pm$2.37}\\
  \cline{2-11}

                                    & TDGC     & 87.16 & 82.44 & 84.74 & \textbf{75.62} & 64.71 & 69.74 & \textbf{73.74} & 63.07 & 67.99\\
                            & +Ours     & \textbf{88.09} & \textbf{84.51} & \textbf{86.26$\pm$2.15} & 74.99 & \textbf{69.92} & \textbf{72.37$\pm$1.94} & 73.45 & \textbf{67.40} & \textbf{70.29$\pm$1.83}\\
  \cline{2-11}
                            & PairGCN  & 88.57 &79.58 & 83.75 & 79.07 & 68.28 & 73.27 & 76.92 & 67.91 & 72.02\\
                            & +Ours  & \textbf{90.55} & \textbf{82.9} & \textbf{86.56$\pm$2.62} & \textcolor{red}{\textbf{79.13}} & \textbf{71.83} & \textbf{75.30$\pm$1.89} & \textcolor{red}{\textbf{78.41}} & \textbf{71.09} & \textbf{74.57$\pm$2.34}\\
  \cline{2-11}
                            &MLL& 86.08 & \textbf{91.91} &\textbf{88.86}  & 73.82 & \textbf{79.12} &76.30 & 77.00 & 72.35 &74.52 \\
                            &+Ours &\textbf{89.64} & 86.13 & 87.85$\pm$1.31 &\textbf{78.34} & 75.89 & \textbf{77.10$\pm$1.03} &\textbf{77.62} & \textbf{73.11} & \textbf{75.29$\pm$1.79} \\
  \cline{2-11}
                &RSN       &86.14  & 89.22& 87.55& 77.27 & 73.98& 75.45& 76.01& 72.19& 73.93\\
                &+Ours       & \textbf{87.11} & \textbf{90.41} & \textbf{88.73$\pm$1.96} & \textbf{77.81} & \textbf{75.63} & \textbf{76.70$\pm$2.27} & \textbf{76.84} & \textbf{75.30} & \textbf{76.06$\pm$2.10} \\

      \hline
    \end{tabular}}
    \caption{Performance comparison in ECPE Chinese corpus for replacing the
    clause-level encoding of all existing approaches with our model (+Ours).
    Except for RANKCP with GAT and JointNN with BiLSTM+attention
    for clause-level encoding, the other approaches above
    adopt BiLSTM to model the clause representation.
    Results significantly show that
    our model is more effective than
    original networks of all existing approaches. }
    \label{tab3}
  \end{table*}

  \begin{table}
    \footnotesize
    \centering
    \resizebox{0.5\linewidth}{!}{
    \begin{tabular}{|c|c|c|c|}
      \hline
      \multirow{2}{*}{model}&\multicolumn{3}{c}{Scores (\%)}\\
      \cline{2-4}
       &P&R&F1\\
      \hline
      CPAM&\textbf{61.95}&78.65&69.29\\
      +Ours&60.72$\pm$1.51&\textbf{84.33$\pm$2.35}&\textbf{72.60$\pm$1.94}\\
      \hline
      IE-CNN&81.86&64.96&66.86\\
      +Ours&\textbf{82.44$\pm$3.81}&\textbf{68.92$\pm$2.27}&\textbf{67.41$\pm$2.56}\\
      \hline
      BERT+&65.03&82.07&72.54\\
      +Ours&\textbf{66.41$\pm$1.24}&\textbf{84.67$\pm$3.34}&\textbf{75.41$\pm$2.14}\\
      \hline
      CL-ECPE$\dagger$&$\diagdown$&$\diagdown$&81.95\\
      +Ours&$\diagdown$&$\diagdown$&\textbf{82.21$\pm$0.79}\\
      \hline
      MGSAG&\textbf{87.17}&77.12&75.21\\
      +Ours&86.82$\pm$1.89&\textbf{81.53$\pm$2.05}&\textbf{76.39$\pm$1.58}\\
      \hline
    \end{tabular}}
    \caption{ The statistics of 5 methods with auxiliary annotation,
    extra data or samples.}
    \label{tabaux}
  \end{table}

  \begin{table*}
    \footnotesize
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{|c|c|ccc|ccc|ccc|}
      \hline
      \multirow{2}{*}{\textbf{Category}} & \multirow{2}{*}{\textbf{Model}} & \multicolumn{3}{c}{emotion extraction(\%)} & \multicolumn{3}{c}{cause extraction(\%)} & \multicolumn{3}{c}{pair extraction(F1(\%))}\\
      \cline{3-11}
      & & P  & R  & F1 &P  & R  & F1 &pos  & neg  & avg\\
      \hline
      \multirow{4}{*}{\thead{with\\RoBERTa}}    & RankCP & \textbf{90.70} & 62.48 & 73.96$\pm$2.98 & \textbf{79.62} & 57.73 & 66.92$\pm$1.44 & 51.45 & 97.06 & 74.26$\pm$1.13\\
      &+Ours & 90.08 & \textbf{64.41} & \textcolor{red}{\textbf{75.08$\pm$3.82}} & 79.00 & \textbf{60.76} & \textbf{67.07$\pm$1.45} & \textbf{52.56} & \textbf{97.14} & \textbf{74.85$\pm$1.46}\\
  \cline{2-11}
                         & ECPE-2D  & 72.17 &41.33 & 52.56$\pm$1.92 & 81.44 & 61.49 & 54.83$\pm$3.65 & 51.07 & 97.11 & 74.09$\pm$2.02\\
     & +Ours  & \textbf{88.57} &\textbf{59.11} & \textbf{70.90$\pm$1.29} & \textbf{82.59} & \textbf{63.81} & \textcolor{red}{\textbf{71.99$\pm$4.35}} & \textbf{53.75} & \textbf{97.37} & \textcolor{red}{\textbf{75.56$\pm$1.87}}\\
      \hline
    \end{tabular}}
    \caption{Performance comparison in RECCON English corpus,
    due to lacking word vectors pre-trained with word2vec toolkit,
    we cancel the experiment without RoBERTa.}
    \label{tab4}
  \end{table*}

  As shown in Table~\ref{tab3},
for evaluating the generality
and effectiveness of our model in ECPE dataset, we embed our model to 8
existing approaches without BERT and 7 approaches with BERT
to summarize the performances on three tasks
(emotion extraction, cause extraction, pair extraction).
In approaches without BERT, the word2vec is adopted
in word embedding, while the pre-training BERT model fine-tuned
is adopted for approaches with BERT.

Overall, our model generally outperforms the traditional clause-level
encoding model (BiLSTM or GAT) in all published approaches. Note that
our model does not adopt any plug-in technology, so these results
demonstrate that our clause-level encoding model is better than all
existing clause-level encoding model.
Besides, our model significantly improves the performance
of graph-based neural network baselines
(i.e.,
RankCP with $3.12\%$ raise without BERT and $3.28\%$ with BERT,
PairGCN with $3.24\%$ raise without BERT and $2.55\%$ with BERT), which
indicates that our model leverages the unique advantages GNNs enjoy
such as the use of part neighboring nodes for aggregation.
And in the graph-agnostic works,
TDGC with $2.25\%$ and $2.3\%$ improvement, Inter-EC with $2.92\%$,
JointNN with $4.65\%$, RSN with $1.8\%$ and $2.13\%$ improvement.
These works entail a more straightforward pair-level encoding
than those with complex downstream models
such as a hybrid encoding for both clause-level and pair-level.
Although our model advances less
in these hybrid encoding works due to the part of superposition
in clause relationship learning, this result reassures that clause
representation orienting to clause-to-clause relationship defined by
ours is effective.

Moreover, to further evaluate the performance in other language
or text genres corpus, as shown in Table\ref{tab4}
we conduct the experiments on the English conversation dataset
named RECCON for the causal emotion entailment task that is
same as ECPE task. Due to the different text genres,
the results in the RECCON are a little
different in increments but demonstrate that the relationship
proposed by ours made our model broadly applicable.

To further indicate the generality of our clause-level encoding method,
we tested the performance in other benchmarks in Table~\ref{tabaux}
by replacing their clause-level encoding model with ours.
From Table~\ref{tabaux}, CPAM+Ours and BERT++Ours perform better than
other benchmarks under the unique conditional annotation dataset their
enjoy. Hence, tagging the causal relationship and adding negative samples
benefit our model to model the clause-to-clause relationship. And CL-ECPE
improves least among all benchmarks. We infer that the adversarial samples
introduce more confounders in clause-to-clause relationship. The same
situation occurs in IE-CNN benchmark which introduces more
relation-agnostic lable types.

\subsection{Ablation Study}

\begin{table}
    \footnotesize
    \centering
    \resizebox{0.5\linewidth}{!}{
    \begin{tabular}{|c|c|c|c|}
      \hline
      \textbf{task}&\textbf{Model} & \textbf{cn: F1(\%)} & \textbf{en: avgF1(\%)} \\
      \hline
      \multirow{5}{*}{C} & Ours & 79.18 &53.1\\
      &-$Loss_{sort}$ &-0.19 &-1.81\\
      &-$Sort$&-0.88 &-3.26\\
      &-$Loss_{sort}$-$Sort$&-1.11&-4.91\\
      &-E-GAT &\textbf{-2.66}& \textbf{-6.44}\\
      \hline
      \multirow{5}{*}{E} & Ours &92.16 &60.92\\
      &-$Loss_{sort}$ &-0.65&-0.63\\
      &-$Sort$&-0.88&-1.12 \\
      &-$Loss_{sort}$-$Sort$&-1&-1.18\\
      &-E-GAT &\textbf{-1.51}&\textbf{-7.07} \\
      \hline
      \multirow{5}{*}{P} & Ours & 76.88&67.33\\
      &-$Loss_{sort}$ &-0.6&-0.86\\
      &-$Sort$&-0.81&-1.33\\
      &-$Loss_{sort}$-$Sort$&-1.07&-1.05\\
      &-E-GAT &\textbf{-2.73}&\textbf{-2.14}\\
      \hline
    \end{tabular}}
    \caption{ Ablation analysis on RANKCP embedded with EA-GAT.
    To facilitate discussion, we show the experiment in
    ECPE corpus with BERT and experiment in RECCON dataset
    without BERT why both have the most obvious gap in the
    4 ablation settings. C, E, and P stand for cause extraction,
    emotion extraction, and pair extraction. Then cn and en denote the
    Chinese corpus and English corpus. }
    \label{tab5}
  \end{table}

  To explore the contribution of E-GAT, Activation Sort,
and $Loss_{sort}$,
3 sets of ablation experiments with the same baseline
on 2 datasets are conducted. In Table\ref{tab5}, we investigate
results under the following cases: removing the $Loss_{sort}$
function; replacing Activation Sort with a $\tanh$ function;
both above cases; and replacing E-GAT with GAT.
One observes that replacing E-GAT leads to the
highest degradation in performance while modifying
$Loss_{sort}$ and Activation Sort yields fewer losses.
And this situation occurs in the two subtasks of
RECCON corpus more sharply.
There are two
reasons we surmise. One is that in the ECPE corpus,
the number of sentences of each sample is as $2.075$ times
as the number in RECCON, where the fewer sentences yield a more
favorable attention distribution. The other one is that
the ratio of ECP to non-ECP in RECCON is $1:10.31$
too much more than $1:174.44$ in ECPE
to not need to widen the gap among
the non-ECPs. In a word, they all indicate that
clause (utterance) relationship is the crucial component that
existing works lack, while Activation Sort is more like a
strategy used for specific data distribution.

\subsection{Case Analysis}\label{sec5.5}

\begin{figure}[hbp]
    \centerline{\includegraphics[width=0.5\linewidth]{case.jpg}}
    \caption{Adjacency matrices of BiLSTM, GAT,  $A^{(0)}$ and $A^{(1)}$}.
    \label{figcase}
  \end{figure}

  To further illustrate how our model improves the performance,
  we inspect the matrix computed from BiLSTM, GAT,
  E-GAT ($A^{(0)}$), and
  EA-GAT ($A^{(1)}$) of the test dataset
  given by the ``RankCP+Ours with BERT'' model in ECPE dataset
  with one visualization shown in Figure\ref{figcase}.

  In BiLSTM, all non-ECPs have almost zero weight except
  the row where the emotion clause is located and the column
  where the cause clause is located.
  It indicates that BiLSTM
  fails to identify whether pairs such as (5,7) (5,8)
  are  correct ECPs.

  In GAT, the relationship weights between non-ECPs
  are different, but
  the weight distribution is concentrated near
  the emotion/cause clauses, and the non-ECPs
  far away from the ECPs still lack effective weight evaluation.

  In $A^{(0)}$ (E-GAT),
  what is different from the above two is, the weight distribution
  is not concentrated near the specific row and column or area,
  but is markedly distinguished to 3 parts via 3 sentences.
  And unlike the above two, the weights of (5,7) and (5,8)
  are significantly lower than the correct ECP such as (5, 4),
  this is because weights of $(5,6)$, $(6,7)$, and $(7,8)$
  are not high, which indicates that there
  is no reverse causal influence from $8$-th clause
  to $5$-th clause.

  In $A^{(1)}$ (EA-GAT), it is obvious that
  there is a series of stepped weight distributions,
  such as $(2,1) (3,2) (4,3) (5,4)$, which matches the fact
  that most clauses are the cause of
  the next clause in the document.
  This result corroborates the theory of
  ~\citet{mann1988rhetorical,marcu2000theory},
  who state that when the clause relationship is aligned with the
  reasoning process
  (in this case, multi-mask attention and activation sort),
  the model
  learns to exploit underlying semantic structure more easily.

\subsection{Sensitivity Aanlysis}

In this section, we further investigate how the different activation
tricks and the layer $l$ would affect the performance. In addition,
we also analyze the sensitivity in between the long document and
short document.

\begin{figure}[hbp]
    \centerline{\includegraphics[width=0.5\linewidth]{layer.jpg}}
  \caption{The results of different layers of EGAT and activation tricks
  under the baseline of RankCP with BERT }.
  \label{figlayer}
\end{figure}

\begin{table}[hbp]
    \footnotesize
    \centering
    \begin{tabular}{|c|c|c|c|c|}
      \hline
     \textbf{category} & \textbf{Approach} & P($\%$) & R($\%$) & F1($\%$) \\
     \hline
     \multirow{2}{*}{$<$15} & RANKCP & 70.96 & 76.33 & 73.55\\
      &Ours& 74.68 & 79.81 & \textbf{77.15} \\

      \hline
      \multirow{2}{*}{$>=$15} & RANKCP & 72.38 & 74.93 & 73.63 \\
      & Ours& 73.94 & 79.76 &\textbf{ 76.72} \\
     \hline
  \end{tabular}
    \caption{ Comparative results for documents with 2 sets of clause numbers}
    \label{tablong}
  \end{table}

In Figure~\ref{figlayer}, we conducted a set of contrasts with the
layers of EGAT ($l$) up to 4 and four activation strick: $\#$2 sort
following Equation~\ref{eqn12}; $\#$1 sort following Equation~\ref{eqn11};
\textit{Sigmoid} activation function; and \textit{tanh} activation function.
From Figure~\ref{figlayer}, there are three observation: (i) Activation Sort($\#$2 sort and $\#$1 sort)
both achieved better performance than activation functions ($sigmoid$ and $tanh$),
which collaborate the standpoint in Section~\ref{subas} that popular activation
functions can not wipe the distance among non-ECPs. (ii) $\#$2 sort achieved
the best performance, which indicate that Equation~\ref{eqn12} is better
designed to highlight the weights of ECPs. (iii) $l=2$ is sufficient
to yeild the most effective clause representation while $l=1$ is not due
to lacking of Activation Sort.

And to analyze the performance of EA-GAT to length of documents,
we divided the entire dataset into approximately equal 2 parts as shown in Table~\ref{tablong},
one for documents with less than 15 clauses
and the other one with more than or equal to 15 clauses.
Then RANKCP is used as the baseline for comparison.
In the category more than 15, RANKCP shows better performance,
while our model increases $3.09\%$ but shows less performance
comparatively to the short documents.
It demonstrates that the EA-GAT network is more suitable for documents
with a short number of clauses.

\section{Results and Discussion}\label{rd}

Results are shown in Table\ref{tab3},\ref{tab4},\ref{tab5} and Figure\ref{fig3},
and we have following observations:

\begin{itemize}
  \item \textbf{EA-GAT is effective. }
  Both under word embedding with BERT and word2vec (without BERT),
  the $F1$ score of pair extraction significantly
  improve by $2.35\%$ and $1.89\%$ averagely.
  Moreover, the $F1$ score of cause extraction improve by
  $1.77\%$ and $2.75\%$ respectively in BERT and without BERT and
  the $F1$ score of emotion extraction improve by $1.17\%$ and $0.89\%$.
  The most improvement occurs in cause extraction which indicates
  that traditional graph-based or recurrence-based model is
  insufficient for clause-to-clause representation learning. And it is
  interesting to see that cause extraction is enhanced more
  in word2vec embedding and emotion extraction more in BERT embedding.
  We consider that emotion extraction depends on word embedding more.
  But they both demonstrate that our model can
  achieve better predictive power by strengthening one of or both subtasks.
  \item \textbf{EA-GAT is general. }
  We detail our comprehensive empirical evaluation from 3 different
  angles to support the generality of our model: (i) our model is integrated with 10 standalone baselines
  with an averagely improve $2.14\%$ in pair extraction
  and $0.98\%$, $2.65\%$ in emotion/cause extraction separately;
  (ii) both in Chinese narrative corpus and
  English conversation corpus, our model has significant advantage
  with $2.1\%$ and $1.03\%$ on average respectively; (iii) from
  the 3 different upstream word embedding methods (word2vec, BERT, and RoBERTa),
  our model substantially surpasses all the baselines with
  $1.89\%$, $2.35\%$, and $1.03\%$ average improvement respectively.
\end{itemize}

While the EA-GAT is generally applicable in ECPE task, there are
some limitations of EA-GAT.

Activation Sort is a radical framework/strategy. We define
it as redical, not bacause it stop the gradient, but because it
maps a clause pair into fixed distribution. Institutionally,
stoping the gradient of a intermediate result absolutely
destroy the whole back propagation. But practically,
as the one of the inputs of the second E-GAT module, it is
affected by two types of loss functions. One is the task loss
including emotion prediction loss, cause prediction loss and
pair prediction loss which measure the predictive power. According
to the input $A^{(0)}$, the word encoding part can learn a
favorable $A^{(0)}$ representation for prediction losses.
Identically, $\widehat{y}^{e}$ and $\widehat{y}^{e}$ is the
input of the second E-GAT also the output of the first E-GAT,
which could lead to that $\widehat{y}^{e}$ and $\widehat{y}^{e}$
lose the purity of thier own prediction results of subtasks.
Such a unnecessarty confilct more substantially occur in sort loss.
Hence, if we used the traditional activation function (i.e., RelU or tanh)
to replace the Sort mapping, we also had to make
$\widehat{y}^{e}$ and $\widehat{y}^{e}$ lose gradients.

What we think that it is imperfect is, for each document, we define
a invariant sequence order for mapping though there are some
alternative sequences in Section 4.3. In this case, each document
has a fixed clause relationship level that should not be the same.
In other words, this is our most concerned problem: how to
construct a clause relationship? The clause relationship, a
unique structure to each document, indicate that which clauses
are irrelevant to the observed clause and which are closely related.
Like a priori knowledge for a neural network to learn the unique
structure among clauses rather than a full-linked structure that
learn a clause representation from all the other clauses. For this
perspective, the clause-level relationship defined by ours is a
breakthrough which fine the grain of clause relationship from the
whole document to sentence. And we attempt to advance the grain
to the single clause via Sort mapping, while the performance of
Sort is not as satisfying as clause-level relationship.

\section{Conclusion}\label{co}

In this paper, we have defined the new and systematic clause-level
relationship for the ECPE task. And to exploit this relationship,
we have developed EA-GAT, a GAT-based model
with multi-mask attention and activation mapping. It produces
an effective clause representation by aggregating the information
from different clause-to-clause relationships and incorporating the
sort position embedding of subtask prediction sequences to achieve grain consistency.
With comprehensive experiments in the two emotion-cause datasets named ECPE and RECCON,
the generality and effectiveness of our model are demonstrated.
Furthermore, we conduct the ablation studies and case analysis to
show that clause-level encoding is an integral contributor to the ECPE task.

\bibliographystyle{elsarticle-num-names}

redpajama_set_name: RedPajamaArXiv


What Amount Of Marijuana Leads To Getting A Misdemeanor Or Felony Charge?
What Is The Difference Between Possession Charge And Intent To Sell Charge?
Is It Recommended To Post A Bond After An Arrest For A Drug Charge?
How Can Someone Get Arrested For A Drug Charge?
How Can You Keep Your Records Clear In A Drug Case?
In Drug Cases, What Options Are Available For Repeat Offenders?
What Are The Pre-Trial Diversion Programs For Drug Cases?
Why Is It Important Not To Violate Terms Of Probation?
Can Your Drug Case Get Dismissed If You Are Not Read Your Miranda Rights?
Is Your Statement Protected By The Miranda Rights In A Drug Case?
Why Miranda Rights Are Important In Drug Cases?
Is Probable Cause Required To Issue An Arrest Warrant For Drug Charges?
Do The Police Have The Right To Make An Arrest For Reasonable Suspicion?
What Rules Are The Policemen Expected To Follow In Search And Seizure?
Does Prescription Drug Dependent In A Drug Possession Case Get Any Leniency?
Is Prosecution Of Prescription Drug Cases Similar To Illegal Drug Cases?
When Does A Drug Case Become A Federal Case?


text:
Canadian Duries
We would like to hear from any Canadian Duries or those with Durie connections who feel they could be included in this section. Please send us their names and some information about their lives and their families. Please send your story and if possible photographs as well. All will be reviewed and may be used in a Newsletter and/or posted on the website. Durie Family Association members will, we hope, be able to provide detail they have on parents, grandparents and ancestors to help us all build up our extended Family Tree.
Canada - The First Durie Migrants - Are you descended from these?
The best-known are the Ontario printing family of Durie, and the descendants of military surgeon Dr William Durie (1779-1862), including William Smith Durie (1813-1885) who had an important career in the Canadian Militia and his son, also William, whose body was famously smuggled back to Canada by his mother and sister after his death at Passchendale during WWI.
Are you one of their descendants? Are you related? Did your ancestors arrive in Canada earlier than Dr William Durie?
We would like any Canadian Duries or those with Durie connections who feel they could be included in this section to send us their names and some information about their lives and their families. Please send your story and if possible photographs as well to webmaster@brucedurie.co.uk. All will be reviewed and may be used in a Newsletter &/or posted on the website. Durie Family Association members will, we hope, be able to provide detail they have on parents, grandparents and ancestors to help us all build up our extended Family Tree.
If you have someone or a history for us to consider adding to the website please send your copy (attached in Word Document or Plain Text only) by e-mail,
accessible from our Contact page. If possible attach photographs in Jpeg format (.jpg). All information will be reviewed and may be used in a Newsletter and/or posted on the website and shared with the family genealogist.
We hope you will also be able to provide details have on parents, grandparents and ancestors to help us build up our extended Family Tree.

redpajama_set_name: RedPajamaCommonCrawl


"Responsibility weighs very heavily on his shoulders": Why shoulders? Why not the hands or legs? Why shouldn't we say "Responsibility weighs very heavily on his little toe"?
The next time you travel by road, observe the various ways in which people keep their balcony gardens. Some would be having the pots neatly arranged, the plants well-pruned, the garden area clean, the plant varieties correctly chosen – basically, a balcony garden that accentuates the beauty and homeliness of the house. On the other hand are gardens where the plants grow wildly on dirty pots; some of them not watered for weeks, dry leaves and stems hanging loosely – looking much like unkempt hair searching for direction in life.
You can judge the amount of energy that will be entering a household by simply observing the way the family takes care of its garden. When you grow a plant, you are actually nurturing a seed of energy. Your plant's growth is directly proportional to the amount of care and effort you put in.
It was a crisp, sunny October morning in the fragrantly nippy, majestic Himalayan land of Binsar. The snow clad Nanda Devi peaks were sparkling like silver jewels, in all their glory as the morning sun washed down upon them. We were languorously waking up to a Diwali morning with some fragrant masala chai in the Eagle's Nest cottage. The girls were rubbing their eyes and gleefully lapping the beauty of the snow clad peaks, from their bunk bed. Laila was still lying curled up against my foot, when Aarshia climbed down the steps of the bunk bed, and cupped her up in her arms.
It was our third morning in Binsar and a morning ritual on this holiday was to go downstairs and sit outside the patio steps and sip into cups of masala chai, while Laila ran around, sniffing the dew covered foliage as the girls gobbled up some chocolate biscuits. Aarshia would finish her hot chocolate after much prodding and thereafter we would all proceed for breakfast to the restaurant. Once again this particular morning we sat at the restaurant balcony, overlooking the serene Himalayas, sunning and savouring it's beauty, while digging into some piping hot puris and alu and some locally made preserves. There were some blueberry and chocolate muffins that we washed down with our coffee. Holidays were meant to feast and forego thoughts of becoming fat and have fun, after all.
Do you lead your life on your terms, or do you just allow your life to lead you?
Do you feel trapped without the trappings of a upper middle-class lifestyle?
Do you find television soap operas dictating how much and when you should spend time with your family?
Have you experienced the 'freedom' of being without a mobile phone?
Do you feel lost and left out when your friends' excitedly discuss about the latest gadgets that they have, that you don't?
If you tick 'yes' to any or all of this, don't worry, you are not alone. Most of us live this complex life. So much so, that we have forgotten how to be simple. We get stuck with jobs that we can't leave because of some or the other loan on our heads. We fall for marketers' tricks, buy more (often on credit) and then spend the rest of our lives, trying to earn to support our 'lifestyle'.
So, is there a way out?
Yes. If you're already addicted to conspicuous consumption, it will take some getting used to. But it is possible to own your own life if you are determined to do it. How? Simplify!! Being simple is an art not many are able to master. The trouble with simple living is that, though it can be joyful, rich, and creative, it isn't simple. And the definition of simplicity is also quite warped.


Once a train depot, this charming lodge has been renovated and converted into accommodations. There are a wide variety of accommodation styles from which families can choose. There is the main lodge, which feature a separate living room with a gas fireplace and a bedroom and bathroom. There are also glamping tents, log cabins, an airstream trailer, a train caboose and cottages. All of the accommodations offer televisions. Cell service and Wi-Fi can be limited here, but Wi-Fi is available in Swinging Bridge Restaurant, the onsite restaurant, and the main Depot Lodge building.
We stayed in Fox Hill. Loved every minute of our stay. Our friends stayed in the lodge and they felt the same way. With the Swinging Bridge across the street you will never leave. Will return very soon and thereafter.
If you enjoy the little different, out of the way spots, this place is for you. The Depot lodge offers all sorts of rental options, from lodge, to caboose, airstream, to glamping. No wifi meant we really got away for a weekend. We stayed in the newest glamping tent, on Pott's creek, and enjoyed it immensely.No AC, but got cool at night. And in the morning they delivered hot coffee, with homemade biscuits and pound cake. It was great!


text:
\section{Introduction}\label{sec:Introduction}
Video makes up the majority of today's Internet traffic. Consequently, this motivates video service companies (e.g. YouTube) to spend substantial effort to control bandwidth usage~\cite{Chen18RQO}. The main remedy deployed is typically video bitrate reduction. However, aggressive bitrate reduction may hurt perceptual visual quality at the same time as both creators and viewers have increasing expectations for streaming media quality. The evolution of new codec technology (HEVC, VP9, AV1) continues to address this bitrate/quality tradeoff.

To measure the quality degradation, numerous quality metrics have been proposed in the last few decades. Some reference-based metrics (like PSNR, SSIM \cite{Wang04SSIM} and VMAF~\cite{Li16VMAF}) have been widely used in the industry.
A common assumption held by most video quality and compression research is that the original video is perfect, and any operation on the original (processing, compression etc.) makes it worse. Most research measures how good the resulting video is by comparing it to the original. However, such an assumption does not hold for most of User Generated Content (UGC) due to the following reasons:

\begin{itemize}
\item \textbf{Non-pristine original} \hspace{1mm} When there are visual artifacts present in the original, it is not clear whether an encoder should be putting in efforts to accurately represent those artifacts. It is necessary to consider the effect that the encoding has on those undesired artifacts, but it is also necessary to consider the effect that the artifacts have on the ability to encode the video effectively.

\item \textbf{Mismatched absolute and reference quality} \hspace{1mm} Using the original as a reference does not always make sense when the original isn't perfect. Quality improvement may be affected by pre/post processing before/after transcoding, but reference-based metrics (e.g. PSNR, SSIM) cannot fairly evaluate the impact of these tools in a compression chain.
\end{itemize}

We created this large scale UGC dataset in order to encourage and facilitate research that considers the practical and realistic needs of video compression and quality assessment in video processing infrastructure.

A major contribution of this work is the analysis of the enormous content in YouTube in a way that illustrates the breadth of visual quality in media worldwide. That analysis leads to the creation of a statistically representative set that is more amenable to academic research and computational resources. We built a large scale UGC dataset(Section~\ref{sec:YouTube UGC Dataset Overview}), and propose (Section~\ref{sec:How the Dataset was Built}) a novel sampling scheme based on features extracted from encoding logs, which achieves high coverage over millions of YouTube videos. Shortcomings of traditional reference-based metrics on UGC are discussed (Section \ref{sec:Challenges on UGC quality assessment}), and we evaluate the dataset with three no-reference metrics: Noise~\cite{Chen16Noise}, Banding~\cite{Wang16Banding}, and Self-reference based LEarning-free Evaluator of Quality (SLEEQ)~\cite{Ghadiyaram17SLEEQ}.

The dataset can be previewed and downloaded from \href{<url>}{\textit{https://media.withyoutube.com/ugc-dataset}}.

\section{Related Work} \label{sec:Related Work}
Some large-scale datasets have already been released for UGC videos, like YouTube-8M~\cite{Haija2016YouTube8M} and AVA~\cite{Gu2018AVA}. However, they only provide extracted features instead of raw pixel data, which makes them less useful for compression research.

Xiph.org Video Test Media~\cite{XiphOrgDataset} is a popular dataset for video compression and it contains around 120 individual video clips (including both pristine and UGC samples). These videos have various resolutions (e.g. SD, HD, and 4K) and multiple content categories (e.g. movie and gaming).

LIVE datasets~\cite{LIVEVideoset,LIVENetflixDataset,LIVEMobileDataset} are also quite popular. All of them contain less than 30 individual pristine clips, along with about 150 distorted versions. The goal of the LIVE datasets is subjective quality assessment. Each video clip in the dataset was assessed by 35 to 55 human subjects.

VideoSet~\cite{VideoSet16} contains 220 5 sec clips extracted from 11 videos. The target here is also quality assessment, but instead of providing Mean Opinion Score (MOS) like the LIVE datasets, it provides the first three Just-Noticeable-Difference (JND) scores collected from more than 30 subjects.

Crowdsourced Video Quality Dataset~\cite{Sinno12018CrowdsourcedQualityDataset} contains 585 10 sec video clips, captured by 80 inexpert videographers. The dataset has 18 different resolutions and a wide range of quality owing to the intrinsic nature of real-world distortions. Subjective opinions were collected from thousands of participants using Amazon Mechanical Turk.

KoNViD-1k~\cite{konvid1k} is another large scale dataset which contains 1200 clips with corresponding subjective scores. All videos are in landscape layout and have resolution higher than $960\times{540}$. They started from a collection of 150K videos, grouping them by multiple attributes like blur, colorfulness etc. The final set was created by a ``fair-sampling'' strategy. The subjective mean opinion scores were gathered through crowdsourcing.

\section{YouTube UGC Dataset Overview} \label{sec:YouTube UGC Dataset Overview}
Our dataset is sampled from YouTube videos with the Creative Commons license. We selected an initial set of 1.5 millions videos belonging to 15 categories annotated by Knowledge Graph~\cite{Singhal12KnowledgeGraph} (as shown in Fig.~ \ref{fig:ugc_category}): Animation, Cover Song, Gaming, HDR, How To, Lecture, Live Music, Lyric Video, Music Video, News Clip, Sports, Television Clip, Vertical Video, Vlog, and VR. The video category is an important feature of our dataset, which allows users to easily explore characteristics of different kinds of videos. For example, Gaming videos usually contain lots of fast motion, while many Lyric videos have still backgrounds. Compression algorithms can be optimized in different ways based on such category information.

Videos within each category are further divided into subgroups based on their resolutions. Resolution is an important feature revealing the diversity of user preferences, as well as the different behaviors of various devices and platforms. So it would be helpful to treat resolution as an independent dimension. In our dataset, we provided $360$P, $480$P, $720$P, $1080$P for all categories (except for HDR and VR) and $4$K for HDR, Gaming, Sports, Vertical Video, Vlog, and VR.

The final dataset contains 1500 video clips, each of 20 sec duration. All clips are in Raw YUV 4:2:0 format with constant framerate. Details of the sampling scheme are discussed in the next section.

\begin{figure}
\centering
\includegraphics[trim={2cm 5.5cm 0cm 2cm}, clip, width=0.46\textwidth]{figures/ugc_category_compressed.pdf}
\caption{All categories in the YouTube UGC dataset.}
\label{fig:ugc_category}
\end{figure}

\section{Dataset Sampling} \label{sec:How the Dataset was Built}
Selecting representative samples from millions of videos is challenging.
Not only is the sheer number of videos a challenge to process and generate features, but also the long duration of some videos (which can be in hours) makes it that much more difficult.
Compared with another large scale dataset, Konvid-1k, this set has videos sampled from a collection that is 10 times larger (1.5M vs. 150K). A video in our dataset can be sampled at any time offset from its original content, instead of taking only the first 30 sec of the original like Konvid-1k. This makes the search space for our dataset 200 times bigger (an average video being 600s long). Due to this huge search space, computing third party metrics (like Blur metric used in Konvid-1k) is resource heavy or even infeasible.

Large scale video compression/transcoding pipelines typically divide long videos into chunks and encode them in parallel. Maintaining the quality consistency among chunk boundaries becomes an issue in practice. Thus, besides the three common attributes (spatial, temporal, and color) suggested in~\cite{Winkler12AnalysisPublicDataset}, we propose the variation of complexity across the video as the fourth attribute that reflects inter-chunk quality consistency. We made the length of video clips in our dataset 20 sec, which is long enough to involve multiple complexity variations. These 20 sec clips could be taken from any segment of a video. For the 5 million hours of videos therefore, there were 1.8 billion putative 20 sec clips.

We used Google's Borg system~\cite{Verma15Borg} to encode each video in the initial collection with FFmpeg H.264 encoder with PSNR stats enabled. The detailed compression settings we used are constant QP 20, fixed GOP size 14 frames with no B frames. Other reasonable settings will also work and bring similar features. The encoder reports on statistics from processing on a per frame basis. That diagnostic output was used to collect the following features over 20 sec clip stepped by 1 sec throughout each video:
\begin{itemize}
\item \textbf{Spatial} In general spatial detail in a frame is correlated with the bits used to encode that frame, when encoded as an Intra frame. Over a 20 sec chunk therefore, we calculate our spatial complexity feature as the average $I$ frame bitrate normalized by the frame area. Fig.~\ref{fig:complexity_comparison_frame}~(a) and Fig.~\ref{fig:complexity_comparison_frame}~(b) are  frames for low and high spatial complexity respectively

\item \textbf{Color} We define color complexity metric as the ratio between the average of mean Sum of Squared Error (SSE) in U and V channels to the mean SSE in Y channel (obtained from PSNR measurements). A high score means complex color variations in the frame (Fig.~\ref{fig:complexity_comparison_frame}~(d)), and a low score usually means a gray image with only luminance changes (Fig.~\ref{fig:complexity_comparison_frame}~(c)).

\item \textbf{Temporal} The number of bits used to encode a $P$ frame is proportional to the temporal complexity of a sequence. However visual material with high spatial complexity (large $I$ frames) tends to have large $P$ frames because small motion compensation errors lead to large residuals. To decouple this effect, we normalize the $P$ frame bits by taking a ratio with the $I$ frame bits, as a fair indicator of temporal complexity.
Fig.~\ref{fig:complexity_comparison}.(a) and Fig.~\ref{fig:complexity_comparison}.(b) show row sum map of videos with low and high temporal complexity, where the $i$th column is the row sum of the $i$th frame. Frequent column color changes in Fig.~\ref{fig:complexity_comparison}.(b) imply fast motion among frames, while small changes in temporal domain leads to homogeneous regions in Fig.~\ref{fig:complexity_comparison}.(a).

\item \textbf{Chunk Variation} To explore quality variation within the video, we measured the standard deviation of compressed bitrates among all 1 sec chunks (also normalized by the frame size). If the scene is static or has a smooth motion, the chunk variation score should be close to 0, and the row sum map has no sudden changes (Fig.~\ref{fig:complexity_comparison}~(c)). Multiple scene changes within a video (common in Gaming videos) will lead to multiple different regions on the row sum map (Fig.~\ref{fig:complexity_comparison}~(d)).
\end{itemize}

\begin{figure}
\centering
\subfigure[Spatial = 0.08]{
\includegraphics[trim={0cm 3.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/complexity_spatial_low_HowTo_1080P-7f4c.pdf}}%
\quad
\subfigure[Spatial = 4.17]{
\includegraphics[trim={0cm 3.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/complexity_spatial_high_Gameplay_1080P-12d4-compressed.pdf}}%
\\
\subfigure[Color = 0.00]{
\includegraphics[trim={0cm 3.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/complexity_color_low_Animation_1080P-209f.pdf}}%
\quad
\subfigure[Color = 1.07]{
\includegraphics[trim={0cm 3.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/complexity_color_high_MusicVideo_1080P-4b61_compressed.pdf}}%
\caption{Various complexity in spatial and color spaces.}
\label{fig:complexity_comparison_frame}
\end{figure}

\begin{figure}
\centering
\subfigure[Temporal = 0.07]{
\includegraphics[trim={0cm 7.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/rowsum_temporal_low_Animation_1080P-3d67.pdf}}%
\quad
\subfigure[Temporal = 1.03]{
\includegraphics[trim={0cm 7.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/rowsum_temporal_high_Vlog_1080P-1609.pdf}}%
\\
\subfigure[Chunk Variation = 0.05]{
\includegraphics[trim={0cm 7.5cm 0cm 0cm}, clip, width=0.2\textwidth]{figures/rowsum_chunk_variation_low_Animation_1080P-5083.pdf}}%
\quad
\subfigure[Chunk Variation = 22.26]{
\includegraphics[trim={0cm 0cm 0cm 7.5cm}, clip, width=0.2\textwidth]{figures/rowsum_chunk_variation_high_Gameplay_1080P-1f29.pdf}}%
\caption{Various complexity in temporal and chunk variation spaces, where the $i$th column is the row sum of the $i$th frame.}
\label{fig:complexity_comparison}
\end{figure}

Each original video and time offset combination forms a candidate clip for the dataset. We sample from this sparse 4-D feature space as described below:
\begin{enumerate}
\item Normalize the feature space $F$ by subtracting $\min(F)$ and dividing with $(\mathrm{percentile}(F, 99) - \min(F))$.
\item For each feature, divide the normalized range $[0, 1]$ uniformly into $N (=3)$ bins, where the last bin can go over 1 and extend up to the maximum value.
\item Permutate all non-empty bins uniformly at random.
\item For current bin, randomly select one candidate clip and add it to the dataset if and only if:
\begin{itemize}
\item The Euclidean distance of feature scores, between this clip and each of the already selected clips is greater than a certain threshold (0.3).
\item The original video that this clip belongs to, doesn't already have another clip in the dataset.
\end{itemize}
\item Move to the next bin and repeat step 4 until desired number of samples are added.
\end{enumerate}
We manually remove mislabeled clips from the 50 selected clips, leaving with 15 to 25 selected clips from each (category, resolution) subgroup. We design the sampling scheme to comprehensively sample along all four feature spaces. As the complexity distribution (normalized) shown in Fig.~\ref{fig:feature_space_coverage}, the feature scores in our sampled set are less spiky than that of the initial set. The sample distribution in color space is left skewed, because many videos with high complexity in other feature spaces are uncolorful. To evaluate the coverage of the sampled set, we divide each pairwise space into $10\times 10$ grid. The percentage of grids covered by the sampled set is shown in Table~\ref{table:Coverage rate for pairwise feature spaces}, where the average coverage rate is 89\%.

\begin{figure}
\centering
\includegraphics[trim={1.5cm 8.5cm 1.5cm 8.5cm}, clip, width=0.48\textwidth]{figures/sampling_histogram.pdf}
\caption{Complexity distributions (normalized) for initial and sampled datasets.}
\label{fig:feature_space_coverage}
\end{figure}

\begin{table}[htbp]
\caption{Coverage rates for pairwise feature spaces. }
\label{table:Coverage rate for pairwise feature spaces}
\centering
\begin{tabular}{|c|c|c|c|}
\hline
    & Temporal   & Color  & Chunk Variation \\
\hline
Spatial      & 93\%  & 90\% & 88\%    \\
\hline
Temporal    &  & 92\% & 88\%    \\
\hline
Color    &   &  & 83\%   \\
\hline
\end{tabular}
\end{table}

\section{Challenges on UGC quality assessment} \label{sec:Challenges on UGC quality assessment}
As mentioned in Section~\ref{sec:Introduction}, most UGC videos are non-pristine, which may confuse traditional reference-based quality metrics. Fig.~\ref{fig:bad_original_sleeq} and Fig.~\ref{fig:bad_original_noise} show two clips (id: Vlog\_720P-343d and Vlog\_720P-670d) from our UGC dataset, as well as their compressed versions (by H.264 with CRF 32). We can see that the corresponding PSNR, SSIM, and VMAF scores are bad, however the original and compressed versions are similar visually. Both the original and compressed versions in Fig.~\ref{fig:bad_original_sleeq} contain significant artifacts, while the compressed version in Fig.~\ref{fig:bad_original_noise} actually has less noise artifacts than the original one. The low reference quality scores are mainly caused by not correctly accounting for artifacts in originals.

\begin{figure}
\centering
\subfigure[Original (SLEEQ = 0.21)]{
\includegraphics[trim={1cm 0cm 0cm 1cm}, clip, width=0.21\textwidth]{figures/bad_original_sleeq_MusicVideo_360P-2a75.pdf}}%
\quad
\subfigure[Compressed (SLEEQ = 0.18)]{
\includegraphics[trim={1cm 0cm 0cm 1cm}, clip, width=0.21\textwidth]{figures/bad_original_sleeq_crf_32_MusicVideo_360P-2a75.pdf}}%
\caption{Misleading reference quality scores, where PSNR, SSIM, and VMAF are 29.02, 0.86, and 58.15.}
\label{fig:bad_original_sleeq}
\end{figure}

\begin{figure}
\centering
\subfigure[Original (noise = 0.32)]{
\includegraphics[trim={4cm 2cm 4cm 2.7cm}, clip, width=0.21\textwidth]{figures/bad_original_noise.pdf}}%
\quad
\subfigure[Compressed (noise = 0.11)]{
\includegraphics[trim={4cm 2cm 4cm 2.7cm}, clip, width=0.21\textwidth]{figures/bad_original_noise_crf_32.pdf}}%
\caption{Misleading reference scores caused by noise artifacts, where PSNR, SSIM, and VMAF are 30.28, 0.63, and 31.43.}
\label{fig:bad_original_noise}
\end{figure}

Although no existing quality metric can perfectly evaluate quality degradation between the original and compressed versions, a possible way is to identify quality issues separately. For example, since we can tell the major quality issue for videos in Fig.~\ref{fig:bad_original_sleeq} is distortions in natural scene, we can compute SLEEQ~\cite{Ghadiyaram17SLEEQ} scores on the original and compressed versions independently, and use their differences to evaluate the quality degradation. In this case, SLEEQ scores for the original and compressed versions are 0.21 and 0.18, respectively, which implies that compression didn't introduce noticeable quality degradation. We can get the same conclusion for videos in Fig.~\ref{fig:bad_original_noise} by applying a noise detector~\cite{Chen16Noise}.

\begin{figure}
\centering
\includegraphics[trim={0cm 10cm 0cm 10cm}, clip, width=0.48\textwidth]{figures/quality_distribution.pdf}
\caption{Distribution of no-reference quality metrics.}
\label{fig:quality_distribution}
\end{figure}

We analyze perceptual quality of our UGC dataset based on existing no-reference metrics. Current evaluation includes three no-reference metrics: Noise, Banding~\cite{Wang16Banding}, and SLEEQ. The first two metrics are artifact-oriented, which can be interpreted as meaningful extents of specific compression issues. The third metric (SLEEQ) is designed to measure compression artifacts on natural scenes. All these metrics have good correlations with human ratings, and outperform other related no-reference metrics. Fig.~\ref{fig:quality_distribution} shows the distribution for the three quality metric scores (normalized to $[0, 1]$ where lower score is better) on our UGC dataset. We can see that heavy artifacts are not detected in most videos, which in some sense tells us that the overall quality of videos uploaded to YouTube is good. Fig.~\ref{fig:quality_metrics_comparison} shows some quality issues within individual categories. For example, Animation videos seem to have more banding artifacts than others, and nature scenes in Vlog tend to contain more artifacts than other categories.
\begin{figure}
\centering
\includegraphics[trim={0cm 8cm 0cm 9cm}, clip, width=0.4\textwidth]{figures/quality_metrics_comparison.pdf}
\caption{Quality comparison for various categories.}
\label{fig:quality_metrics_comparison}
\end{figure}

\section{Conclusion} \label{sec:Conclusion}
This paper introduced a large scale dataset for UGC videos, which highly represents the videos uploaded to YouTube. A novel sampling scheme was proposed to extract features from millions of video samples, and we investigated complexity distribution as well as quality distribution for videos across 15 categories.
The difficulties of UGC quality assessment were cited. An important open question remains regarding how to evaluate quality degradation caused by compression for UGC (non-pristine reference). We hope this dataset can inspire and facilitate research on practical requirements of video compression and quality assessment.

\bibliographystyle{IEEEbib}


 I believe therapy is not about telling a stranger all your problems, but sharing your pain, triumphs and laughter with someone who you connect and feel comfortable with, but also with someone who can provide a nonjudgmental space to help you sort through it all.
I have been gracious to have experience in many different fields of wellness and mental health to develop a systemic perspective on healing. I believe that relationships with others is the key to healing and growing.
I created Relational Balance as a way for couples to grow together in a world that continually tries to draw them apart. When we are working together, I hope to help my clients grow in their relationship and create a new, healthy experience together.
Compassion and humor are found at my core and I share these with all my clients. My down-to-earth approach to life allows me to develop real connections with my clients. My number one goal as a therapist is create a laid-back environment for my clients to explore their relationships in comfortably.
My therapy is strength-based, focusing on repairing communicating, increasing physical and emotional connection, and repairing past wounds. I have completed Level 1 Training in The Gottman Method and often utilize Gottman principles in my practice.
My greatest hope, is that by working together, you will feel secure, validated, and empowered to create the life in which feels authentic to you.
In my free time I enjoy time laughing with my husband and family. I have two dogs that have loads of personality who also enjoy cuddling up and watching football with me on the weekends.


Now nearing its 60th printing in English and translated into nineteen languages, Michael E. Porter's Competitive Strategy has transformed the theory, practice, and teaching of business strategy throughout the world. Electrifying in its simplicity -- like all great breakthroughs -- Porter's analysis of industries captures the complexity of industry competition in five underlying forces.
Find out how you're doing before you access our marketing intelligence and competitive analysis tools and resources. Take the Competitive Intelligence Assessment. The assessment addresses key aspects of competitive intelligence and analysis. It will help you determine the current state of your competitive intelligence and analysis practices and processes. Your answers will indicate specific areas where you may benefit from improvements. And, it will also help guide your decisions about your future competitive intelligence and analysis needs and investments. You will be given a score upon completion. The assessment includes 84 questions and should take approximately 15 minutes to complete.
This section includes a variety of competitive analysis tools to help you assess your organization's competitive position and improve it. The tools range from those designed for a quick assessment and overview to those designed to conduct a detailed analysis of your organization's and your competitor's competitive position on wide range of factors, including: financial performance, marketing, sales and other factors.
Use the Competitive Marketing Performance Evaluator to score a competitive organization's performance on each of twenty-six key activities and assess their level of importance in achieving its overall goals. You can compare your organization and up to four competitors or any five organizations.
Is a scan of the internal and external environment factors affecting your business; and, as a tool, it is an important step in the strategic planning process. This tool is used to conduct a comprehensive S.W.O.T. (strengths, weaknesses, opportunities, and threats). It will enable you to assess your organization's internal and external market facing strengths and weaknesses, and comparing them to those of your competitors.
Is a comprehensive (61 questions) assessment of your organization and your competitors in the areas of management, finance, sales and marketing, distribution, product and pricing, quality, support and service. The tool allows you to rate each competitive factor as to importance and then rate your organization and each competitor on actual performance against that factor.
Enables you to compare your organization's and up to five of your competitor's performance in six key categories. The table and chart that are produced will give you an idea of where there are gaps in your competitors' market strengths as they relate to your company's own strengths.
Will give you a quick visual representation of where your business ranks verses your competition. Where is your business located on this continuum? How does your business rank in these areas? The tool will give you an overall ranking based on the settings you choose. Quickly compare your business to your competitors' in 13 key areas.
Will help you quickly assess your organization's competitive position and determine the major issues or problems you must address to improve the competitive position of your organization.
The competitive positioning mapping tool will help you compare your capabilities against those of five competitors based on customer buying criteria.
Intellicomp CIS (Competitive Intelligence System) is an information management system designed to hold a wide spectrum of strategic and tactical information about your competitors and your plans and programs for dealing with the competition. It has an easy-to-use interface that provides access to all of your competitive information in a straight-forward manner.


The Katarinahissen lift has been taking people up from Slussen to the Katarinaberget district on Södermalm since 1883. The views are even better with a drink in hand at Eriks Gondolen, the restaurant and bar at the top.
Even bored teenagers have a sharp intake of breath walking into this darkened museum. The Vasa, the seemingly invincible, richly decorated pride of the Swedish navy, sank on its maiden voyage in 1628. A film show takes you through its recovery and room settings give an idea of the lives of long-dead sailors. The star, the reconstructed Vasa, sits in its shadowy home, maintaining a dignified remoteness.
You're in good company in this three-storey 18th-century inn. The Swedish Academy, the august body that nominates the winner of the annual Nobel Prize for Literature, meets here every Thursday. The wooden floors slope and the candles flicker but the restaurant has a growing reputation for its modern take on traditional Swedish cooking.
Children love the miniature reproductions of life from the 17th century through to today in elaborately furnished dolls' houses. Adults are equally captivated by the table settings, state bedchambers, folk art and textiles in this gigantic toybox of a museum.
A costumed guide leads the way through the a palace overflowing with 17th-century Flemish paintings, 500 years' worth of silverware and humble umbrellas and pots and pans. Every item was catalogued by the excessively wealthy obsessive, Countess Hallwylska. Her extraordinary stately home remains just as she left it on her death in 1930.
Indoor swimming doesn't get much grander than in the Art Nouveau Centralbadet spa of 1904. Afterwards, relax over a coffee in the peaceful outdoor garden beside the ornamental pool and fountain.
Hop on a restored Number 7 vintage tram to leafy Djurgården. Trams run regularly in the summer from Norrmalmstorg and back. There's even a café tram for coffee on the go.
Opera and ballet lovers flock to Stockholm's Opera House during the season for the Royal Ballet and Opera. During the summer break, it's the turn of the guides to entertain with stories like the assassination of King Gustav III at a masked ball in 1792. It inspired Verdi's Un Ballo in Maschera.
Take the hour-long boat ride to Drottningholm Palace to see why the royal family prefers to live here. The waterside estate has a palace full of lavish galleries, Rococco gardens, a rolling English-style park and a Chinese pavilion. Spend a magical summer evening at the baroque theatre. It still uses the original wooden stage machinery.
This small, invariably crowded venue with its faded black-and-white photographs feels more 1930s than 21st century. There's no entrance fee - just drop some cash into the basket as it's handed round.