text:
\section{Introduction}
\vspace{-.25em}

Neural models that exploit word embeddings have recently achieved impressive results on text classification tasks \cite{goldberg2015primer}. Feed-forward Convolutional Neural Networks (CNNs), in particular, have emerged as a relatively simple yet powerful class of models for text classification \cite{kim2014convolutional}

These neural text classification models have tended to assume a standard supervised learning setting in which instance labels are provided. Here we consider an alternative scenario in which we assume that we are provided a set of \emph{rationales}
\cite{zaidan2007using,zaidan2008,McDonnell16-hcomp} in addition to instance labels, i.e., sentences or snippets that support the corresponding document categorizations. Providing such rationales during manual classification is a natural interaction for annotators, and requires little additional effort \cite{settles2011,McDonnell16-hcomp}. Therefore, when training new classification systems, it is natural to acquire supervision at both the document and sentence level, with the aim of inducing a better predictive model, potentially with less effort.

Learning algorithms must be designed to capitalize on these two types of supervision. Past work (Section \ref{section:related-work}) has introduced such methods, but these have relied on linear models such as Support Vector Machines (SVMs)~\cite{joachims1998text}, operating over sparse representations of text. We propose a novel CNN model for text classification that exploits both document labels and associated rationales.

Specific contributions of this work as follows. (1) This is the first work to incorporate rationales into neural models for text classification. (2) Empirically, we show that the proposed model uniformly outperforms relevant baseline approaches across five datasets, including previously proposed models that capitalize on rationales \cite{zaidan2007using,marshall2016robotreviewer} and multiple baseline CNN variants, including a CNN equipped with an attention mechanism. We also report state-of-the-art results on the important task of automatically assessing the risks of bias in the studies described in full-text biomedical articles \cite{marshall2016robotreviewer}. (3) Our model naturally provides \emph{explanations} for its predictions, providing interpretability.

We have made available online both a Theano\footnote{\url{https://github.com/yezhang-xiaofan/Rationale-CNN}} and a Keras implementation\footnote{\url{https://github.com/bwallace/rationale-CNN}} of our model.

\section{Related Work}
\label{section:related-work}

\subsection{Neural models for text classification}
\newcite{kim2014convolutional} proposed the basic CNN model we describe below and then build upon in this work. Properties of this model were explored empirically in \cite{zhang2015}. We also note that \newcite{zhang2016mgnc} extended this model to jointly accommodate multiple sets of pre-trained word embeddings. Roughly concurrently to Kim, \newcite{johnson2014effective} proposed a similar CNN architecture, although they swapped in one-hot vectors in place of (pre-trained) word embeddings. They later developed a semi-supervised variant of this approach \cite{johnson2015semi}.

In related recent work on Recurrent Neural Network (RNN) models for text, \newcite{tang2015document} proposed using a Long Short Term Memory (LSTM) layer to represent each sentence and then passing another RNN variant over these. And \newcite{yang2016hierarchical} proposed a hierarchical network with two levels of attention mechanisms for document classification. We discuss this model specifically as well as attention more generally and its relationship to our proposed approach in Section \ref{section:AT-CNN}.

\subsection{Exploiting \emph{rationales}}
In long documents the importance of sentences varies; some are more central than others. Prior work has investigated methods to measure the relative importance sentences~\cite{ko2002automatic,murata2000japanese}. In this work we adopt a particular view of sentence importance in the context of document classification. In particular, we assume that documents comprise sentences that directly support their categorization. We call such sentences \emph{rationales}.

The notion of rationales was first introduced by \newcite{zaidan2007using}. To harness these for classification, they proposed modifying the Support Vector Machine (SVM) objective function to encode a preference for parameter values that result in instances containing manually annotated rationales being more confidently classified than `pseudo'-instances from which these rationales had been stripped. This approach dramatically outperformed baseline SVM variants that do not exploit such rationales. \newcite{yessenalina2010automatically} later developed an approach to \emph{generate} rationales.

Another line of related work concerns models that capitalize on \emph{dual supervision}, i.e., labels on individual \emph{features}. This work has largely involved inserting constraints into the learning process that favor parameter values that align with \emph{a priori} feature-label affinities or rankings \cite{druck2008learning,mann2010generalized,small2011constrained,settles2011}. We do not discuss this line of work further here, as our focus is on exploiting provided rationales, rather than individual labeled features.

\section{Preliminaries: CNNs for text classification}
\label{section:preliminaries}

\begin{figure}[ht!]
\centering
\includegraphics[width=0.35\textwidth]{single-CNN.pdf}
\caption{A toy example of a CNN for sentence classification. Here there are four filters, two with heights 2 and two with heights 3, resulting in feature maps with lengths 6 and 5 respectively.}
\label{Single_CNN}
\end{figure}

We first review the simple one-layer CNN for sentence modeling proposed by \newcite{kim2014convolutional}. Given a sentence or document comprising $n$ words $w_1,w_2,$...,$w_n$, we replace each word with its $d$-dimensional pretrained embedding, and stack them row-wise, generating an instance matrix $A\in\mathbb{R}^{n\times d}$.

We then apply convolution operations on this matrix using multiple linear filters, these will have the same width $d$ but may vary in \emph{height}. Each filter thus effectively considers distinct $n$-gram features, where $n$ corresponds to the filter height. In practice, we introduce multiple, redundant features of each height; thus each filter height might have hundreds of corresponding instantiated filters. Applying filter $i$ parameterized by $W_i\in\mathbb{R}^{h_i\cdot d}$ to the instance matrix induces a \emph{feature map} $f_i\in\mathbb{R}^{n-h_i+1}$. This process is performed by sliding the filter from the top of the matrix (the start of the document or sentence) to the bottom. At each location, we apply element-wise multiplication between filter $i$ and sub-matrix $A[j:j+h_i-1]$, and then sum up the resultant matrix elements. In this way, we induce a vector (feature map) for each filter.

We next run the feature map through an element-wise non-linear transformation. Specifically, we use the \emph{Rectified Linear Unit}, or ReLU~\cite{krizhevsky2012imagenet}. We extract the maximum value $o_i$ from each feature map $i$ (\emph{1-max pooling}).

Finally, we concatenate all of the features $o_i$ to form a vector representation $\mathbf{o}\in\mathbb{R}^{|F|}$ for this instance, where $|F|$ denotes the total number of filters. Classification is then performed on top of $\mathbf{o}$, via a softmax function. Dropout~\cite{srivastava2014dropout} is often applied at this layer as a means of regularization. We provide an illustrative schematic of the basic CNN architecture just described in Figure \ref{Single_CNN}. For more details, see~\cite{zhang2015}.

This model was originally proposed for sentence classification \cite{kim2014convolutional}, but we can adapt it for document classification by simply treating the document as one long sentence. We will refer to this basic CNN variant as \textbf{\textit{CNN}} in the rest of the paper. Below we consider extensions that account for document structure.

\section{Rationale-Augmented CNN for Document Classification}
\label{section:ra-CNN}

We now move to the main contribution of this work: a rationale-augmented CNN for text classification. We first introduce a simple variant of the above CNN that models document structure (Section \ref{section:doc-CNN}) and then introduce a means of incorporating rationale-level supervision into this model (Section \ref{section:ra-cnn-ss}). In Section \ref{section:AT-CNN} we discuss connections to attention mechanisms and describe a baseline equipped with one, inspired by \newcite{yang2016hierarchical}.

\subsection{Modeling Document Structure}
\label{section:doc-CNN}

Recall that rationales are snippets of text marked as having supported document-level categorizations. We aim to develop a model that can exploit these annotations during training to improve classification. Here we achieve this by developing a hierarchical model that estimates the probabilities of individual sentences being rationales and uses these estimates to inform the document level classification.

As a first step, we extend the CNN model above to explicitly account for document structure. Specifically, we apply a CNN to each individual sentence in a document to obtain sentence vectors independently. We then sum the respective sentence vectors to create a document vector.\footnote{We also experimented with taking the average of sentence vectors, but summing performed better in informal testing.
}
As before, we add a softmax layer on top of the document-level vector to perform classification. We perform regularization by applying dropout both on the individual sentence vectors and the final document vector. We will refer to this model as \textbf{\textit{Doc-CNN}}. Doc-CNN forms the basis for our novel approach, described below.

\subsection{RA-CNN}
\label{section:ra-cnn-ss}

In this section we present the Rationale-Augmented CNN (\textit{\textbf{RA-CNN}}). Briefly, RA-CNN induces a document-level vector representation by taking a \emph{weighted} sum of its constituent sentence vectors. Each sentence weight is set to reflect the estimated probability that it is a rationale in support of the most likely class. We provide a schematic of this model in Figure \ref{DS_CNN}.

\begin{figure*}[t]
\includegraphics[width=1.0\textwidth]{rob-cnn3.pdf}
\caption{A schematic of our proposed Rationale-Augmented Convolution Neural Network (RA-CNN). The sentences comprising a text are passed through a sentence model that outputs probabilities encoding the likelihood that sentences are neutral or a (positive or negative) rationale. Sentences likely to be rationales are given higher weights in the global document vector, which is the input to the document model.}
\label{DS_CNN}
\end{figure*}

RA-CNN capitalizes on both sentence- and document-level supervision. There are thus two steps in the training phase: \emph{sentence level} training and \emph{document level} training. For the former, we apply a CNN to each sentence $j$ in document $i$ to obtain sentence vectors $\mathbf{x}_\text{sen}^{ij}$. We then add a softmax layer parametrized by $\textbf{W}_\text{sen}$; this takes as input sentence vectors. We fit this model to maximize the probabilities of the observed rationales:

\begin{equation}
   p(y_{\text{sen}}^{ij}=k; \textbf{E},\textbf{C},\textbf{W}_\text{sen}) = \frac{\text{exp}(\textbf{W}_\text{sen}^{(k)T}\mathbf{x}_{\text{sen}}^{ij})}{\sum_{k=1}^{K_\text{sen}}\text{exp}(\textbf{W}_\text{sen}^{(k)T}\mathbf{x}_{\text{sen}}^{ij})}
   \label{Eq:sen}
\end{equation}

\noindent Where $y_{\text{sen}}^{ij}$ denotes the rationale label for sentence $j$ in document $i$, $K_\text{sen}$ denotes the number of possible classes for sentences, \textbf{E} denotes the word embedding matrix, \textbf{C} denotes the convolution layer parameters, and $\textbf{W}_\text{sen}$ is a matrix of weights (comprising one weight vector per sentence class)

\begin{comment}
\begin{equation}
    J(\textbf{E},\textbf{C},\textbf{W}_\text{sen}) = -\sum_{i=1}^{B}\sum_{k=1}^K 1\{y_\text{sen}^{ij}=k\}\text{log}p(y_\text{sen}^{ij}=k)
    \label{Eq:sen-loss}
\end{equation}
\end{comment}

In our setting, each sentence has three possible labels ($K_\text{sen}=3$). When a rationale sentence appears in a positive document,\footnote{All of the document classification tasks we consider here are binary, although extension of our model to multi-class scenarios is straight-forward.} it is a \textit{positive rationale}; when a rationale sentence appears in a negative document, it is a \textit{negative rationale}. All other sentences belong to a third, neutral class: these are \textit{non-rationales}.
We also experimented with having only two sentence classes: \emph{rationales} and \emph{non-rationales}, but this did not perform as well as explicitly maintaining separate classes for rationales of different polarities.

We train an estimator using the provided rationale annotations, optimizing over \{$\textbf{E},\textbf{C},\textbf{W}_\text{sen}$\} to minimize the categorical cross-entropy of sentence labels. Once trained, this sub-model can provide conditional probability estimates regarding whether a given sentence is a positive or a negative rationale, which we will denote by $p_\text{pos}$ and $p_\text{neg}$, respectively.

We next train the document-level classification model. The inputs to this are vector representations of documents, induced by summing over constituent sentence vectors, as in Doc-CNN. However, in the RA-CNN model this is a weighted sum. Specifically, weights are set to the estimated probabilities that corresponding sentences are rationales in the most likely direction. More precisely:
\begin{equation}
    \mathbf{x}_{\text{doc}}^i = \sum_{j=1}^{N_i}\mathbf{x}_{\text{sen}}^{ij} \cdot \text{max}\{p_{\text{pos}}^{ij},p_{\text{neg}}^{ij}\}
\end{equation}

\noindent Where $N_i$ is the number of sentences in the $i$th document.
The intuition is that sentences likely to be rationales will have greater influence on the resultant document vector representation, while the contribution of neutral sentences (which are less relevant to the classification task) will be minimized.

The final classification is performed by a softmax layer parameterized by $\mathbf{W}_\text{doc}$; the inputs to this layer are the document vectors. The $\mathbf{W}_\text{doc}$ parameters are trained using the document-level labels, $y_\text{doc}^i$:

\begin{equation}
    p(y_\text{doc}^i=k;\mathbf{E},\mathbf{C},\mathbf{W}_\text{doc}) = \frac{\text{exp}(\mathbf{W}_\text{doc}^{(k)T}\mathbf{x}_{\text{doc}}^{i})}{\sum_{k=1}^{K_\text{doc}}\text{exp}(\mathbf{W}_\text{doc}^{(k)T}\mathbf{x}_{\text{doc}}^i)}
\end{equation}

\noindent where $K_{\text{doc}}$ is the cardinality of the document label set. We optimize over parameters to minimize cross-entropy loss (w.r.t. the document labels).

\begin{comment}
\begin{align*}
    J(\mathbf{E},\mathbf{C},\mathbf{W}_\text{doc}) = -\sum_{i=1}^{M}\sum_{k=1}^K 1\{y_\text{doc}^i=k\}\\
    *\text{log}p(y_\text{doc}^i=k)
\end{align*}
\end{comment}

We note that the sentence- and document-level models share word embeddings $\mathbf{E}$ and convolution layer parameters $\mathbf{C}$, but the document-level model has its own softmax parameters $\mathbf{W}_\text{doc}$. When training the document-level model, $\mathbf{E}$, $\mathbf{C}$ and $\mathbf{W}_\text{doc}$ are fit, but we hold $\mathbf{W}_\text{sen}$ fixed.

The above two-step strategy can be equivalently described as follows. We first estimate $\mathbf{E}$, $\mathbf{C}$ and $\mathbf{W}_\text{sen}$, which parameterize our model for identifying rationales in documents. We then move to fitting our document classification model. For this we initialize the word embedding and convolution parameters to the $\mathbf{E}$ and $\mathbf{C}$ estimates from the preceding step. We then directly minimize the document level classification objective, tuning $\mathbf{E}$ and $\mathbf{C}$ and simultaneously fitting $\mathbf{W}_\text{doc}$.

Note that this sequential training strategy differs from the \emph{alternating} training approach commonly used in multi-task learning~\cite{collobert2008unified}. We found that the latter approach does not work well here, leading us to instead adopt the cascade-like feature learning approach~\cite{collobert2008unified} just described.

\begin{comment}
In our case, it's quite different from multi-task training, since we only have one final task: classifying documents, and classifying sentences into whether a rationale or not is just one intermediate step. We treat it as a pre-training task before document classification, and experimentally, it performs better than alternative training.
\end{comment}

One nice property of our model is that it naturally provides explanations for its predictions: the model identifies rationales and then categorizes documents informed by these. Thus if the model classifies a test instance as \emph{positive}, then by construction the sentences associated with the highest $p_{\text{pos}}^{ij}$ estimates are those that the model relied on most in coming to this disposition. These sentences can of course be output in conjunction with the prediction. We provide concrete examples of this in Section \ref{section:predicted-rationales}.

\subsection{Rationales as `Supervised Attention'}
\label{section:AT-CNN}

One may view RA-CNN as a supervised variant of a model equipped with an \emph{attention mechanism} \cite{bahdanau2014neural}. On this view, it is apparent that rather than capitalizing on rationales directly, we could attempt to let the model learn which sentences are important, using \emph{only} the document labels. We therefore construct an additional baseline that does just this, thereby allowing us to assess the impact of learning directly from rationale-level supervision.

Following the recent work of \newcite{yang2016hierarchical}, we first posit for each sentence vector a hidden representation $\mathbf{u}^{ij}_{\text{sen}}$. We then define a sentence-level \emph{context vector} $\mathbf{u}_s$, which
we multiply with each $\mathbf{u}^{ij}_{\text{sen}}$ to induce a weight $\alpha_{ij}$. Finally, the document vector is taken as a weighted sum over sentence vectors, where weights reflect $\alpha$'s. We have:

\begin{equation}
    \mathbf{u}^{ij}_{\text{sen}} = \text{tanh}(\mathbf{W}_s\mathbf{x}^{ij}_{\text{sen}}+b_s)
\end{equation}
\begin{equation}
    \alpha_{ij} = \frac{\text{exp}(\mathbf{u}_s^T \mathbf{u}^{ij}_{\text{sen}})}{\sum_j^{N_i} \text{exp}(\mathbf{u}_s^T\mathbf{u}^{ij}_{\text{sen}})}
\end{equation}

\begin{equation}
    \mathbf{x}_{\text{doc}}^i = \sum_j^{N_i}\alpha_{ij}\mathbf{x}_{\text{sen}}^{ij}
\end{equation}
where $\mathbf{x}_{\text{doc}}^i$  again denotes the document vector fed into a softmax layer, and $\mathbf{W}_s$, $\mathbf{u}_s$ and $b_s$ are learned during training. We will refer to this attention-based method as \emph{AT-CNN}

\section{Datasets}
\label{section:datasets}

We used five text classification datasets to evaluate our approach in total. Four of these are biomedical text classification datasets (\ref{section:risk-of-bias-data}) and the last is a collection of movie reviews (\ref{section:movie-review-dataset}). These datasets share the property of having recorded rationales associated with each document categorization. We summarize attributes of all datasets used in this work in Table \ref{Dataset}.

\subsection{Risk of Bias (RoB) Datasets}
\label{section:risk-of-bias-data}

We used a collection \emph{Risk of Bias} (RoB) text classification datasets, described at length elsewhere~\cite{marshall2016robotreviewer}. Briefly, the task concerns assessing the reliability of the evidence presented in full-text biomedical journal articles that describe the conduct and results of randomized controlled trials (RCTs). This involves, e.g., assessing whether or not patients were properly blinded as to whether they were receiving an active treatment or a comparator (such as a placebo). If such blinding is not done correctly, it compromises the study by introducing statistical bias into the treatment efficacy estimate(s) derived from the trial.

A formal system for making bias assessments is codified by the Cochrane Risk of Bias Tool \cite{higgins2011cochrane}. This tool defines multiple \emph{domains}; the risk of bias may be assessed in each of these. We consider four domains here. (1) Random sequence generation (RSG): were patients were assigned to treatments in a truly random fashion? (2) Allocation concealment (AC): were group assignments revealed to the person assigning patients to groups (so that she may have knowingly or unknowingly) influenced these assignments? (3) Blinding of Participants and Personnel (BPP): were all trial participants and individuals involved in running the trial blinded as to who was receiving which treatment? (4) Blinding of outcome assessment (BOA): were the parties who measured the outcome(s) of interest blinded to the intervention group assignments? These assessments are somewhat subjective. To increase transparency, researchers performing RoB assessment therefore record rationales (sentences from articles) supporting their assessments.

\begin{comment}
The task is to identify whether a clinical trial is bias.
RoB dataset has four domains: Random sequence generation (RSG), Allocation concealment (AC), Blinding of participants and personnel (BPP) and Blinding of outcome assessment (BOA). We treat these four domains as four independent datasets, so equivalently, we have four datasets. We'll run 5-fold cross validation (CV) on these four datasets.
\end{comment}

\begin{table
\centering
\small
 \begin{tabular}{c c c c c}

& \emph{N} &\emph{\#sen}& \emph{\#token} &\emph{\#rat} \\

 \hline
 RSG & 8399 & 300 & 9.92 & 0.31 \\

 AC & 11512 & 297 &9.87 & 0.15 \\

 BPP & 7997 & 296 & 9.95 & 0.21 \\

 BOA & 2706 & 309 & 9.92 & 0.2 \\
 \hline
 MR & 1800 & 32.6 & 21.2 & 8.0 \\

\end{tabular}
\caption{Dataset characteristics. \emph{N} is the number of instances, \emph{\#sen} is the average sentence count, \emph{\#token} is the average token per-sentence count and \emph{\#rat} is the average number of rationales per document.}

\vspace{-1.0em}
\label{Dataset}
\end{table}

\subsection{Movie Review Dataset}
\label{section:movie-review-dataset}

We also ran experiments on a movie review (MR) dataset with accompanying rationales. \newcite{pang2004sentimental} developed and published the original version of this dataset, which comprises 1000 positive and 1000 negative movie reviews from the Internet Movie Database (IMDB).\footnote{\url{http://www.imdb.com/}} ~\newcite{zaidan2007using} then augmented this dataset by adding rationales corresponding to the binary classifications for 1800 documents, leaving the remaining 200 for testing. Because 200 documents is a modest test sample size, we ran 9-fold cross validation on the 1800 annotated documents (each fold comprising 200 documents).
The rationales, as originally marked in this dataset, were sub-sentential snippets; for the purposes of our model, we considered the entire sentences containing the marked snippets as rationales.

\section{Experimental Setup}
\label{section:experiment-setup}

\subsection{Baselines}
\label{section:baselines}

We compare against several baselines to assess the advantages of directly incorporating rationale-level supervision into the proposed CNN architecture.
We describe these below.

\vspace{.25em}

\noindent {\bf SVMs}. We evaluated a few variants of linear Support Vector Machines (SVMs). These rely on sparse representations of text. We consider variants that exploit uni- and bi-grams; we refer to these as \emph{uni-SVM} and \emph{bi-SVM}, respectively. We also re-implemented the rationale augmented SVM (\emph{RA-SVM}) proposed by ~\newcite{zaidan2007using}, described in Section \ref{section:related-work}.

For the RoB dataset, we also compare to a recently proposed multi-task SVM (MT-SVM) model developed specifically for these RoB datasets~\cite{marshall2015automating,marshall2016robotreviewer}. This model exploits the intuition that the risks of bias across the domains codified in the aforementioned Cochrane RoB tool will likely be correlated. That is, if we know that a study exhibits a high risk of bias for one domain, then it seems reasonable to assume it is at an elevated risk for the remaining domains. Furthermore, \newcite{marshall2016robotreviewer} include rationale-level supervision by first training a (multi-task) \emph{sentence-level} model to identify sentences likely to support RoB assessments in the respective domains. Special features extracted from these predicted rationales are then activated in the \emph{document-level} model, informing the final classification. This model is the state-of-the-art on this task.

\vspace{.25em}

\noindent {\bf CNNs}. We compare against several baseline CNN variants to demonstrate the advantages of our approach. We emphasize that our focus in this work is \textbf{not} to explore how to induce generally `better' document vector representations -- this question has been addressed at length elsewhere, e.g., ~\cite{le2014distributed,jozefowicz2015empirical,tang2015document,yang2016hierarchical}.

Rather, the main contribution here is an augmentation of CNNs for text classification to capitalize on rationale-level supervision, thus improving performance and enhancing interpretability. This informed our choice of baseline CNN variants: standard CNN \cite{kim2014convolutional}, Doc-CNN (described above) and AT-CNN (also described above) that capitalizes on an (unsupervised) attention mechanism at the sentence level, described in Section \ref{section:AT-CNN}.\footnote{We also experimented briefly with LSTM and GRU (Gated Recurrent Unit) models, but found that simple CNN performed better than these. Moreover, CNNs are relatively robust and less sensitive to hyper-parameter selection.}

\begin{comment}
\subsection{SVM}
For SVM classifier, we use linear kernel and Stochastic Gradient Descent (SGD) as the optimization method.

\noindent
\mathbf{RA-SVM} We follow exactly the same procedure in \cite{zaidan2007using}: we use an independent development set to optimize $\mu$, $C$, and $C_{contrast}$ in their paper.

\subsection{CNN}
\noindent
\mathbf{CNN} We use 3 different filter heights: 3, 4 and 5, and for each filter height, we use 100 feature maps. We set the dropout rate as 0.5, and max norm constraint on the softmax layer as 3.

\noindent
\mathbf{Doc-CNN} The setup is the same as CNN, except we use 20 feature maps for each filter height, so we end up with 60 feature maps for each sentence in the document, as well as for the document representation.
\end{comment}

\subsection{Implementation/Hyper-Parameter Details}
\label{section:configurations}

\begin{table*}
\centering
\small
 \begin{tabular}{l c c c c}

Method & RSG & AC & BPP & BOA \\[0.5ex]
\hline
\emph{Uni-SVM} & 72.16 & 72.81  & 72.80 & 65.85 \\
\emph{Bi-SVM} & 74.82 &73.62 & 75.13& 67.29 \\
\emph{RA-SVM} & 72.54 &74.11 & 75.15 & 66.29  \\
\emph{MT-SVM} & 76.15 & 74.03 & 76.33 & 67.50 \\
\emph{CNN} & 72.50 (72.22, 72.65) & 72.16 (71.49, 72.93)& 75.03 (74.16, 75.44) & 63.76 (63.12, 64.15) \\
\emph{Doc-CNN} & 72.60 (72.43, 72.90) &72.92 (72.19, 73.48)  & 74.24 (74.03, 74.38) & 63.64 (63.23, 64.37)\\
\emph{AT-CNN} & 74.14 (73.40, 74.58) & 73.66 (73.12, 73.92)& 74.29 (74.09, 74.74) & 63.34 (63.21, 63.49)\\
\emph{RA-CNN} & \textbf{77.42 (77.33, 77.59)} & \textbf{76.14 (75.89, 76.29)} & \textbf{76.47 (76.15, 76.75)} & \textbf{69.67 (69.33, 69.93)}\\
\hline
\emph{Human} & 85.00 & 80.00 & 78.10 & 83.20  \\
\hline
\end{tabular}
\caption{Accuracies on the four RoB datasets. Uni-SVM: unigram SVM, Bi-SVM: Bigram SVM, RA-SVM: Rationale-augmented SVM \protect\cite{zaidan2007using}, MT-SVM: a multi-task SVM model specifically designed for the RoB task, which also exploits the available sentence supervision  \protect\cite{marshall2016robotreviewer}. We also report an estimate of human-level performance, as calculated using subsets of the data for each domain that were assessed by two experts (one was arbitrarily assumed to be correct). We report these numbers for reference; they are not directly comparable to the cross-fold estimates reported for the models.}
\label{RoB_perf}
\vspace{-.75em}
\end{table*}

 \begin{comment}
 \begin{figure
\includegraphics[width=0.5\textwidth]{RoB_results-new-3.pdf}
\caption{Accuracies on the four RoB datasets; RA-CNN uniformly outperforms the baselines.
\vspace{-1em}
\label{figure:ROB_result}
\end{figure}
\end{comment}}

\noindent {\bf Sentence splitting.} To split the documents from all datasets into sentences for consumption by our Doc-CNN and RA-CNN models, we used the Natural Language Toolkit (NLTK)\footnote{http://www.nltk.org/api/nltk.tokenize.html} sentence splitter.

\vspace{.25em}

\noindent {\bf SVM-based models.} We kept the 50,000 most frequently occurring features in each dataset. For estimation we used SGD. We tuned the $C$ hyper-parameter using nested development sets. For the RA-SVM, we additionally tuned the $\mu$ and $C_{contrast}$ parameters, as per \newcite{zaidan2007using}.

\vspace{.25em}

\noindent {\bf CNN-based models.}  For all models and datasets we initialized word embeddings to pre-trained vectors fit via Word2Vec. For the movie reviews dataset these were 300-dimensional and trained on Google News.\footnote{https://code.google.com/archive/p/word2vec/} For the RoB datasets, these were 200-dimensional and trained on biomedical texts in PubMed/PubMed Central \cite{moendistributional}.\footnote{http://bio.nlplab.org/}

Training proceeded as follows. We first extracted all sentences from all documents in the training data. The distribution of sentence types is highly imbalanced (nearly all are neutral). Therefore, we downsampled sentences before each epoch, so that sentence classes were equally represented. After training on sentence-level supervision, we moved to document-level model fitting. For this we initialized embedding and convolution layer parameters to the estimates from the preceding sentence-level training step (though these were further tuned to optimize the document-level objective).

For RA-CNN, we tuned the dropout rate (range: 0-.9) applied at the sentence vector level on each training fold (using a subset of the training data as a validation set) during the document level training phase.
Anecdotally, we found this has a greater effect than the other model hyperparameters, which we thus set after a small informal process of experimentation on a subset of the data. Specifically,  we fixed the dropout rate at the document level to 0.5, and we used 3 different filter heights: 3, 4 and 5, following \cite{zhang2015}. For each filter height, we used 100 feature maps for the baseline CNN, and 20 for all the other CNN variants.

For parameter estimation we used ADADELTA \cite{zeiler2012adadelta}, mini-batches of size 50, and an early stopping strategy (using a validation set).

\section{Results and Discussion}
\label{section:results}

\subsection{Quantitative Results}

For all CNN models, we replicated experiments 5 times, where each replication constituted 5-fold and 9-fold CV
respectively the RoB and the movies datasets, respectively.
We report the mean and observed ranges in accuracy across these 5 replications for these models, because attributes of the model (notably, dropout) and the estimation procedure render model fitting stochastic~\cite{zhang2015}. We do not report ranges for SVM-based models because the variance inherent in the estimation procedure is much lower for these simpler, linear models.

Results on the RoB datasets and the movies dataset are shown in Tables \ref{RoB_perf} and Table \ref{Movie_Perf}, respectively.
RA-CNN consistently outperforms all of the baseline models, across all five datasets. We also observe that CNN/Doc-CNN do not necessarily improve over the results achieved by SVM-based models, which prove to be strong baselines for longer document classification.
This differs from previous comparisons in the context of classifying shorter texts. In particular, in previous work  ~\cite{zhang2015} we observed that CNN outperforms SVM uniformly on sentence classification tasks (the average sentence-length in these datasets was about 10). In contrast, in the datasets we consider in the present paper, documents often comprise hundreds of sentences, each in turn containing multiple words. We believe that it is in these cases that explicitly modeling which sentences are most important will result in the greatest performance gains, and this aligns with our empirical results

Another observation is that AT-CNN does often improve performance over vanilla variants of CNN (i.e., without attention), especially on the RoB datasets, probably because these comprise longer documents. However, as one might expect, RA-CNN clearly outperforms AT-CNN by exploiting rationale-level supervision directly.
And by exploiting rationale information directly, RA-CNN is able to consistently perform better than baseline CNN and SVM model variants. Indeed, we find that RA-CNN outperformed MT-SVM on all of the RoB datasets, and this was accomplished without exploiting cross-domain correlations (i.e., without multi-task learning).

\begin{comment}
And we note that even though we only coarsely tune the hyperparameters of RA-CNN, and very carefully tune SVM and RA-SVM across a very broad range, RA-CNN can still consistently achieve the best result.
\end{comment}

\begin{comment}
\begin{figure*}
\centering
\begin{minipage}{0.4\textwidth}
\centering
\includegraphics[width=\textwidth]{ROB_results-new-3.pdf}
\caption{Accuracies achieved by different models across the four RoB datasets.}
\label{figure:ROB_result}
\end{minipage}\hfill
\begin{minipage}{0.5\textwidth}
\centering
\includegraphics[width=\textwidth]{movie_results-new-3.pdf}
\caption{Accuracies on the movies dataset. RA-CNN again outperforms all baselines.}

\label{figure:movie_result}
\end{minipage}
\end{figure*}
\end{comment}

\begin{comment}
We find that the dropout rate on the sentence vector is an important hyperparameter. The sentence level classification might sometimes give non-rationales very high probability, making the real rationale have a very low probability, possibly close to 0. When we weight each sentence in the document level training, the non-rationale might dominate the training, which will propagate the error. The dropout on the sentence vector could prevent this happening.
\end{comment}

\begin{table}
\small
\centering
 \begin{tabular}{l c}
 Method & Accuracy \\
 \hline
\emph{Uni-SVM} & 86.44\\
\emph{Bi-SVM} & 86.94\\
\emph{RA-SVM} & 88.89\\
\emph{CNN} & 85.59 (85.27, 86.17) \\
\emph{Doc-CNN} & 87.14 (86.70, 87.60)\\
\emph{AT-CNN} & 86.69 (86.28, 87.17) \\
\emph{RA-CNN} & \textbf{90.43 (90.11, 91.00)}\\
\hline
\end{tabular}
\caption{Accuracies on the movie review dataset.}

\label{Movie_Perf}
\vspace{-1.25em}
\end{table}

\begin{comment}
\begin{figure
\centering
\includegraphics[width=0.5\textwidth]{movie_results-new-3.pdf}
\vspace{-1em}
\caption{Accuracies achieved on the movies dataset. RA-CNN outperforms all baseline models.
\label{figure:movie_result}
\vspace{-1em}
\end{figure}
\end{comment}

\vspace{-.5em}
\subsection{Qualitative Results: Illustrative Rationales}
\vspace{-.1em}
\label{section:predicted-rationales}

\begin{comment}

\begin{table*}[ht]
\small

\label{table:rob-rationales}
\begin{tabular}{l p{5.75cm} p{8.25cm}}
Domain & Positive rationale example & Negative rationale example\\
\hline
 BPP & \emph{Patients, cardiologists and nurses were all blinded to treatment assigned} &  \emph{Injuries were done by weekly reviews of the log-in sheets at the troop medical clinic and review of the medical records to determine the diagnosis of any subject seen}  \\
 \hline
 RSG & \emph{The order of the two laser procedures and the eye treated were first randomized using a random number table.} &  \emph{The selection of the restoration material was made by randomization after the patient had agreed to participate. }  \\
 \hline
BOA & \emph{The study was performed double blind.} &  \emph{As the present study is retrospective, there is a risk that the woman did not properly recall how and what they experienced at the time the information about their risks and about the surgical procedures was presented to them.}    \\
\hline
AC & \emph{The two companies designated as either the control or intervention group were assigned randomly before basic training.} & \emph{Patients were randomized into two groups using sequentially numbered, sealed envelopes.}      \\
\end{tabular}
\caption{Rationales automatically extracted for (correctly classified) test articles in the RoB datasets.}
\label{table:rationales}
\end{table*}

\end{comment}

In addition to realizing superior classification performance, RA-CNN also provides \emph{explainable} categorizations. The model can provide the highest scoring rationales (ranked by max$\{p_\text{pos},p_\text{neg}\}$) for any given target instance, which in turn -- by construction -- are those that most influenced the final document classification.

For example, a sample positive rationale supporting a correct designation of a study as being at low risk of bias with respect to blinding of outcomes assessment reads simply \emph{The study was performed double blind}. An example rationale extracted for a study (correctly) deemed at high risk of bias, meanwhile, reads \emph{as the present study is retrospective, there is a risk that the woman did not properly recall how and what they experienced ...}.

Turning to the movie reviews dataset, an example rationale extracted from a glowing review of `Goodfellas' (correctly classified as positive) reads \emph{this cinematic gem deserves its rightful place among the best films of 1990s}. While a rationale extracted from an unfavorable review of `The English Patient' asserts that \emph{the only redeeming qualities about this film are the fine acting of Fiennes and Dafoe and the beautiful desert cinematography}.

In each of these cases, the extracted rationales directly support the respective classifications. This provides direct, meaningful insight into the automated classifications, an important benefit for neural models, which are often seen as opaque.

\begin{comment}
\begin{table*}[h!]

\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: The study was performed double blind.

\textbf{Negative}: As the present study is retrospective, there is a risk that the woman did not properly recall how and what they experienced at the time the information about their risks and about the surgical procedures was presented to them.
 }%
}
\caption{The top estimated rationale in a positive and a negative test document from BOA dataset.}
\end{table*}

\begin{table*}[h!]
\scriptsize
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{+}: The order of the two laser procedures and the eye treated were first randomized using a random number table.

\textbf{-}: The selection of the restoration material was made by randomization after the patient had agreed to participate.
}%
}
\caption{The top estimated rationale in a positive and a negative test document from RSG dataset.}
\end{table*}

\begin{table*}[h!]
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: The two companies designated as either the control or intervention group were assigned randomly before basic training.

\textbf{Negative}: Patients were randomized into two groups using sequentially numbered, sealed envelopes.
}%
}
\caption{The top estimated rationale in a positive and a negative test document from AC dataset.}

\end{table*}

\begin{table*}[h!]
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: Patients, cardiologists and nurses were all blinded to treatment assigned.

\textbf{Negative}: Injuries were done by weekly reviews of the log-in sheets at the troop medical clinic and review of the medical records to determine the diagnosis of any subject seen.
}%
}
\caption{BPP}
\end{table*}
\end{comment}

\begin{comment}
\begin{table*}[ht]
\scriptsize
\centering
\subfloat[Blinding of Participants and Personnel]{
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{+} Patients, cardiologists and nurses were all blinded to treatment assigned.

\textbf{-} Injuries were done by weekly reviews of the log-in sheets at the troop medical clinic and review of the medical records to determine the diagnosis of any subject seen.
}%
}}
\qquad
\subfloat[RSG]{
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: The order of the two laser procedures and the eye treated were first randomized using a random number table.

\textbf{Negative}: The selection of the restoration material was made by randomization after the patient had agreed to participate.
}%
}
}
\qquad
\subfloat[BOA]{
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: The study was performed double blind.

\textbf{Negative}: As the present study is retrospective, there is a risk that the woman did not properly recall how and what they experienced at the time the information about their risks and about the surgical procedures was presented to them.
 }%
}}
\qquad
\subfloat[AC]{
\noindent\fbox{%
    \parbox{\textwidth}{
\textbf{Positive}: The two companies designated as either the control or intervention group were assigned randomly before basic training.

\textbf{Negative}: Patients were randomized into two groups using sequentially numbered, sealed envelopes.
}%
}
}
\caption{Top estimated rationale in a positive test document and in a negative test document from the four datasets in RoB dataset.}
\label{rationale_RoB}
\end{table*}
\end{comment}

\begin{comment}
\begin{table*}[!ht]
  \begin{threeparttable}
    \begin{tabularx}{\textwidth}{lX}
     \midrule
      \textbf{Positive} &  1. because of this achievement, and also because of the great influence on future filmmakers, this cinematic gem deserves its rightful place among the best films of 1990s.\\

&2. this film was goodfellas , epic black comedy which is today considered as one of the best and most influential films of 1990s.\\

&3. another essential element of goodfellas is large but superb collection of great acting talents .\\

&4. goodfellas is an excellent example of a film that represents work of a film genius in his full glory.\\

&5. joe pesci deserved his "oscar" for supporting role of homicidal psychopath tommy , and his lines , including ad-libs , probably represent the most memorable element of the film .\\
      \midrule
      \textbf{Negative}                  & 1. other than these , the english patient is full of worthless scenes of boredom and wastes entirely too much film .  \\
      & 2. great acting , music and cinematography were nice , but too many dull sub-plots and characters made the film hard to follow .\\
      & 3. maybe if it were an hour less , it may have been tolerable , but 2 hours and 40 minutes of talking is too much to handle .\\
      & 4. lengthy and lousy are two words to describe the boring drama the english patient .\\
      & 5. the only redeeming qualities about this film are the fine acting of fiennes and dafoe and the beautiful desert cinematography .\\
      \bottomrule
    \end{tabularx}
  \end{threeparttable}
  \caption{Top five estimated rationales in a positive and a negative test document from Movie Review dataset.}
  \label{rationale_Movie}
\end{table*}

\end{comment}

\begin{comment}
\section{Related Work}
\label{section:related-work}

The first work to popularize Convolutional Neural Networks (CNNs) for text (specifically, sentence) classification was \newcite{kim2014convolutional}. We review the basic architecture in Section \ref{section:preliminaries}.

\cite{zhang2016mgnc} uses the same model, but combines different pre-trained word embeddings.
For document classification, \cite{johnson2014effective} uses one-hot vector to encode text, and train CNN on this representation. \cite{tang2015document} first uses Long Short Term Memory (LSTM) to represent each sentence, and build another layer of variant Recurrent Neural Network (RNN) on top of this. \cite{yang2016hierarchical} builds a hierachical attention network with two levels of attention mechanisms.

\cite{zaidan2007using} provides a movie review dataset with rationale annotation, and proposes a new Support Vector Machine (SVM) framework that can incorporate rationales information during training. Their model greatly boosts the performance compared with situations without rationales.
\cite{small2011constrained} proposes a Constrained Weighed SVM (CS-SVM) model, which utilizes the ranked features provided by the expert.

\end{comment}

\section{Conclusions}
\label{section:conclusions}

We developed a new model (RA-CNN) for text classification that extends the CNN architecture to directly exploit \emph{rationales} when available. We showed that this model outperforms several strong, relevant baselines across five datasets, including vanilla and hierarchical CNN variants, and a CNN model equipped with an attention mechanism. Moreover, RA-CNN automatically provides explanations for classifications made at test time, thus providing interpretability.

Moving forward, we plan to explore additional mechanisms for exploiting supervision at lower levels in neural architectures. Furthermore, we believe an alternative approach may be a hybrid of the AT-CNN and RA-CNN models, wherein an auxiliary loss might be incurred when the attention mechanism output disagrees with the available direct supervision on sentences.

\section*{Acknowledgments}

Research reported in this article was supported by the National Library of Medicine (NLM) of the National Institutes of Health (NIH) under award number R01LM012086. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This work was also made possible by the support of the Texas Advanced Computer Center (TACC) at UT Austin.

redpajama_set_name: RedPajamaArXiv


text:
\section{Introduction}

The investigation of the baryon spectrum and the baryon couplings
from experimental data are two of the most important issues in
hadronic physics and they are attracting much attention. Both on the
experimental and theoretical sides, the nucleon excited states below
$2.0$ GeV have been extensively studied~\cite{pdg2012}. However, the
current information for the properties of states around or above
$2.0$ GeV is scarce~\cite{pdg2012}. On the other hand in this region
of energies, many theoretical approaches (constituent
quark~\cite{capstick2000} and chiral
unitary~\cite{Gamermann:2011mq,Sarkar:2009kx,Oset:2009vf,PavonValderrama:2011gp,Sun:2011fr})
predict predicted {\it missing $N^*$ states}, which have not been so
far observed. Hence, the study of the possible role played by the
$2.0$ GeV region nucleon resonances in the available accurate data
is timely and could shed light into the complicated dynamics that
governs the high excited nucleon spectrum.

The associate $K \Lambda(1520)$ photon and hadronic production
reactions might be adequate to study the $N^*$ resonances around
$2.0$ GeV, as long as they have significant couplings to the
$K\Lambda(1520)$ pair. This is because the $K\Lambda(1520)$ is a
pure isospin $1/2$ channel and the threshold is about $2.0$ GeV.
Besides, these reactions require the creation of an $\bar{s}s$ quark
pair. Thus, a thorough and dedicated study of the strangeness
production mechanism in these reactions has the potential to gain a
deeper understanding of the interaction among strange hadrons and
also on the nature of the nucleon resonances.

Recently, there have been several measurements for the $\gamma p \to
K^+\Lambda(1520)$
reaction~\cite{leps1,leps2,Wieland:2011zz,Moriya:2013hwg}. These
data suggest that there is a sizeable contribution of total and
differential cross sections from the nucleon resonances with masses
around $2.1$ GeV. On the theoretical side, in addition to the
contributions from $K$ and $K^*$ exchange in the $t-$channel and the
contact term, the contributions from the nucleon states, including
the $N^*(2120)$, in the
$s-$channel~\cite{nam2,xiejuan,hejun,nam3,xieenjuan} and the
$\Lambda(1115)$ pole in the $u-$channel have been
studied~\cite{xieenjuan}. The theoretical results show that when the
contributions from the $N^*(2120)$ resonance and the $\Lambda(1115)$
are taken into account, the current experimental
data~\cite{leps1,leps2,Moriya:2013hwg} can be well described. Thus,
it is becoming clear that the current experimental data for
$\Lambda(1520)$ photoproduction favor the existence of the
$N^*(2120)$ resonance, and that these measurements can be used to
further constrain its properties. On the other hand, based on the
results of $\gamma p \to K^+\Lambda(1520)$ reaction, the
$K\Lambda(1520)$ production in the $pp \to p K^+\Lambda(1520)$ and
$\pi^- p \to K^0 \Lambda(1520)$ hadronic processes are also studied,
paying an special attention to the contributions from the
$N^*(2120)$ resonance~\cite{xieliu}. In the present work, we will
review the main results from these theoretical studies.

\section{Study on $\gamma p \to K^+ \Lambda(1520)$ reaction}

For the $\gamma p \to K^+ \Lambda(1520)$ reaction, the differential
cross section, in the center of mass frame (c.m.), and for a
unpolarized photon beam reads,
\begin{eqnarray}
\frac{d\sigma}{d(\cos\theta_{\rm c.m.})}  &=& \frac{|\vec{k}_1^{\rm
\,\, c.m.}||\vec{p}_1^{\rm \,\,c.m.}|}{4\pi}\frac{M_N
M_{\Lambda^*}}{(W^2 - M_N^2)^2} \sum_{s_p, s_\Lambda^*,\lambda}
|T|^2, \nonumber
\end{eqnarray}
with $W$ the invariant mass of the $\gamma p$ pair. Besides,
$\vec{k}_1^{\rm \,\, c.m.}$ and $\vec{p}_1^{\rm \,\, c.m.}$ are the
photon and $K^+$ meson c.m. three-momenta, and $\theta_{\rm c.m.}$
is the  $K^+$ polar scattering angle. The invariant scattering
amplitudes are defined as
\begin{equation}
-iT_i=\bar u_\mu(p_2,s_{\Lambda^*}) A_i^{\mu \nu} u(k_2,s_p)
\epsilon_\nu(k_1,\lambda), \nonumber
\end{equation}
where $u_\mu$ and $u$ are dimensionless Rarita-Schwinger and Dirac
spinors for final $\Lambda(1520)$ and the initial proton,
respectively, while $\epsilon_\nu(k_1,\lambda)$ is the photon
polarization vector. Besides, $s_p$ and $s_{\Lambda^*}$ are the
baryon polarization variables. The sub-index $i$ stands for the
contact, $t-$channel $K^-$ exchange, contact, $s-$channel nucleon
pole and $N^*$ resonance terms (depicted in Fig.~1 of
Ref.~\cite{xiejuan}) and the $u-$channel $\Lambda(1115)$
contribution (see Fig.~2 of Ref.~\cite{xieenjuan}).

\begin{figure}
\begin{center}
\includegraphics[scale=0.23]{dcs65.eps}
\includegraphics[scale=0.23]{dcs75.eps}
\includegraphics[scale=0.23]{dcs85.eps}
\includegraphics[scale=0.23]{dcs95.eps}%
\caption{The $\gamma p \to K^+ \Lambda(1520)$ differential
$d\sigma/d(\cos\theta_{\rm c.m.})$ cross sections at forward $K^+$
angles compared with the LEPS data. Dashed and dotted lines show the
contributions from the background ($t-$channel $K^-$ exchange,
$s-$channel nucleon pole, and contact term) and $N^*(2120)$
resonance terms, respectively, while the solid line displays the
full result. For this latter curve we also show the statistical 68\%
confidence level (CL) band.}
\label{dcs-leps}%
\end{center}
\end{figure}

In Fig.~\ref{dcs-leps}, the theoretical calculations of the
differential cross section $d\sigma/d(\cos\theta_{\rm c.m.})$ as a
function of the photon beam energy $E_{\gamma}$ are shown. These
predictions are also compared to the experimental data taken from
LEPS collaboration~\cite{leps2}. The contributions from different
mechanisms of the model are separately shown. We see that the bump
structure in the differential cross section could be fairly well
described thanks to a significant contribution from the $N^*(2120)$.
It is worth to mention that in this calculation the contribution
from the $u-$channel $\Lambda(1115)$ pole has been neglected, since
its contribution is small at forward $K^+$ angels.

\begin{figure}
\begin{center}
\includegraphics[scale=0.75]{dcsbackwardtcs.eps}
\caption{The $\gamma p \to K^+ \Lambda (1520)$ differential cross
section (left panel) and total cross section (right panel) as a
function of the photon beam energy $E_{\gamma}$. Left panel:
black-dash-dotted curves show the $u-$channel $\Lambda (1115)$
contribution, while the red-solid lines show the full contributions.
Right panel: blue-dashed and green-dotted lines show the
contributions from the background and $N^*(2120)$ resonance terms,
respectively, while the red-solid line displays the results from the
full model. In both panels, we display the 68\% CL bands.}
\label{dcs-deg}%
\end{center}
\end{figure}

Next in Fig.~\ref{dcs-deg}, we show theoretical results for
differential cross sections at large $K^+$ angles and the total
cross section. Here, we also compare our predictions with the
experimental data from Refs.~\cite{leps1,Moriya:2013hwg}. In this
case, we pay attention not only to the contribution from $s-$channel
$N^*(2120)$ resonance but also to that from the $u-$channel
$\Lambda$ pole, since the $u-$channel contribution at the backward
angles could be important. As can be seen in the Fig.~\ref{dcs-deg},
the theoretical calculations provides a fair description of these
backward $K^+$ angular data thanks to the contribution from the
$\Lambda$ pole term in the $u-$channel. Furthermore, for the total
cross section, due to an important contribution from the
photo-excitation of the $N^*(2120)$ resonance and its subsequent
decay into a $\Lambda(1520)K^+$ pair, the theoretical results can
describe the CLAS data~\cite{Moriya:2013hwg} very well (see right
panel of Fig.~\ref{dcs-deg}). This mechanism is also important for
the bump structure in the LEPS differential cross section at forward
$K^+$ angles discussed in Fig.~\ref{dcs-leps}. Thus, one can
definitely take advantage of the important role played by this
resonant mechanism in the LEPS and CLAS data to better constrain
some of the $N^*(2120)$ properties, starting from its mere
existence.

\section{Study on $\pi^- p \to K^0 \Lambda(1520)$ and $pp \to pK^+ \Lambda(1520)$ reactions}

Since the $N^*(2120)$ resonance played an important role in the
$\gamma p \to K^+\Lambda(1520)$ reaction as discussed above, it may
also have important contributions to the $\pi^- p \to K^0
\Lambda(1520)$ and $pp \to pK^+ \Lambda(1520)$ reactions, which has
also been studied in Ref.~\cite{xieliu}.

For the $\pi^- p \to K^0 \Lambda(1520)$ reaction, the differential
cross section in the $\rm c.m.$ frame can be expressed as

\begin{equation}
{d\sigma \over d{\rm cos}\theta}={1\over 32\pi s}{
|\vec{p_3}^{\text{c.m.}}| \over |\vec{p_1}^{\text{c.m.}}|} \left (
{1\over 2}\sum_{s_{\Lambda^*},s_p}|T|^2 \right ). \nonumber
\end{equation}
In the equation above $\theta$ denotes the angle of the outgoing
$K^0$ relative to beam direction in the $\rm c.m.$ frame, while
$\vec{p_1}^{\text{c.m.}}$ and $\vec{p_3}^{\text{c.m.}}$ are the
three-momenta of the initial $\pi^-$ and final $K^0$ mesons,
respectively. The total invariant scattering amplitude $T$ is given
by,
\begin{equation}
T=T_s + T_t + T_u  + T_R \, ,
\end{equation}
where the contributions from the $s-$channel nucleon pole,
$t-$channel $K^*$ exchange, $u-$channel $\Sigma^+$ and $s-$channel
$N^*(2120)$ terms are considered.

With the parameters for the $N^*(2120)K\Lambda(1520)$ strong
couplings that were obtained from the $\gamma p \to K^+
\Lambda(1520)$ reaction, the role of the $N^*(2120)$ resonance in
the $\pi^- p \to K^0\Lambda(1520)$ reaction has been investigated.
Theoretical results for the total $\pi^- p \to K^0 \Lambda(1520)$
cross section are shown in Fig.~\ref{piptcs}, and compared with the
data taken from Ref.~\cite{pipdata}. The solid lines represent the
full results, while the contributions from the $s-$channel nucleon
pole, $t-$channel $K^*$ exchange, $u-$channel $\Sigma^+$ and
$s-$channel $N^*(2120)$ terms are shown by the dashed, dotted,
dot-dashed, and dash-dot-dotted lines, respectively. We see that we
can describe the experimental data of total cross sections quite
well, while the $s-$channel nucleon pole and $N^*(2120)$ resonance
and also the $u-$channel $\Sigma^+$ exchange give the dominant
contributions below $\sqrt{s}=2.4$ GeV. The $t-$channel $K^*$
exchange diagram gives a minor contribution.

\begin{figure}[htbp]
\begin{center}
\includegraphics[scale=0.45]{piptcs.eps}
\caption{Total cross sections vs the invariant mass
$s^{\frac{1}{2}}$ for the $\pi^- p \to K^0 \Lambda(1520)$ reaction.
The curves stand for the contributions from the $s-$channel nucleon
pole (dashed), $t-$channel $K^*$ (dotted), $u-$channel $\Sigma^+$
term (dash-dotted) and the $s-$channel $N^*(2080)$ terms
(dash-dot-dotted) and the total contribution (solid).}
\label{piptcs}
\end{center}
\end{figure}

For the $pp \to pK^+ \Lambda(1520)$ reaction, the total cross
section versus the beam momentum (${\rm p_{lab}}$) of the proton is
calculated by using a Monte Carlo multi-particle phase space
integration program. The results for beam momentum ${\rm p_{lab}}$
from just above the production threshold $3.59$ GeV to $5.0$ GeV are
shown in Fig.~\ref{pptcs}. The dashed, dotted, and dash-dotted lines
stand for contributions from nucleon pole, $\Sigma^+$ pole and
$N^*(2120)$ resonance, respectively. The total contribution is shown
by the solid line. From Fig.~\ref{pptcs}, we mechanisms see that the
contribution from the $u-$channel $\Sigma^+$ exchange is dominant
very close to threshold, but, when the beam energy increases, the
contributions from the $s-$channel nucleon pole and the $N^*(2120)$
resonance turn to be very important.

\begin{figure}[htbp]
\begin{center}
\includegraphics[scale=0.5]{pptcs.eps}
\caption{Total cross section vs proton beam momentum ${\rm p_{lab}}$
for the $pp \to pK^+\Lambda(1520)$ reaction. The dashed, dotted, and
dash-dotted lines stand for contributions from nucleon pole,
$\Sigma^+$ pole and $N^*(2080)$ resonance, respectively. The total
contribution is shown by the solid line.} \label{pptcs}
\end{center}
\end{figure}

It is worth to note that our predictions for the total $pp \to
pK^+\Lambda(1520)$ cross section, at ${\rm p_{lab}} = 3.65$ GeV, is
$0.01 \mu b$, which is $20$ times smaller than the experimental
upper limit of $0.2 \mu b$ as measured by the COSY-ANKE
Collaboration~\cite{zychorcosy}. This shows that our model
predictions are consistent with the experimental results. Moreover,
the total cross section of the $pp \to pK^+\Lambda(1520)$ reaction
has been also measured with HADES~\cite{l1520data} at GSI for a
kinetic energy ${\rm T_p} = 3.5$ GeV (corresponding to ${\rm
p_{lab}} = 4.34$ GeV). The result is $5.6\pm1.1\pm0.4^{+1.1}_{-1.6}
~\mu$b, as shown in Fig.~\ref{pptcs}, this is to be compared with
our theoretical result, $11.5 ~\mu$b. However, if we modify the cut
off parameters $\Lambda_{\pi}$ and $\Lambda^*_{\pi}$ from 1.3 GeV to
1.0 GeV, we get $\sigma = 5.45 ~\mu$b, which would be in agreement
with the experimental data. However, it does not make sense to fit
only one data point. So we still keep $\Lambda_{\pi} =
\Lambda^*_{\pi}$ = 1.3 GeV as used in many previous
works~\cite{xiezou}. We should also mention that, in the present
calculation, we did not include the $\Lambda(1520) p$
final-state-interaction (FSI), which can increase the results even
by a factor of 10 at the very near threshold region, similarly to
the important role played by $\Lambda p$ FSI in the $pp \to
pK^+\Lambda$ reaction~\cite{Xie:2011me}. We ignore this effect
because there are no experimental data on this reaction and also
very scarce information about the $\Lambda(1520) p$ FSI.

Furthermore, the corresponding momentum distributions of the final
proton and $K^+$ meson, the $K\Lambda(1520)$ invariant mass
spectrum, and also the Dalitz Plot for the $pp \to pK^+
\Lambda(1520)$ reaction at beam momentum ${\rm p_{lab}} = 3.67$ GeV,
which is accessible for DISTO Collaboration~\cite{baledisto}, are
calculated and shown in Fig.~\ref{plab367}(a),
Fig.~\ref{plab367}(b), Fig.~\ref{plab367}(c), and
Fig.~\ref{plab367}(d), respectively. The dashed lines are just phase
space distributions, while, the solid lines are full results from
our model. From Fig.~\ref{plab367}, we see that even at ${\rm
p_{lab}} = 3.67$ GeV, there is a clear bump in the $K\Lambda(1520)$
invariant mass distribution, which is produced by the contribution
of the $N^*(2120)$ resonance.

\begin{figure}[htbp]
\begin{center}
\includegraphics[scale=0.7]{pl1520k367.eps}
\caption{Momentum distribution (arbitrary units), invariant mass
spectrum (arbitrary units), and Dalitz Plot for the $pp \to pK^+
\Lambda(1520)$ reaction at beam momentum ${\rm p_{lab}} = 3.67$ GeV
compared with the phase space distribution. The dashed lines are
phase space distributions, while the solid lines display full
results from our model.} \label{plab367}
\end{center}
\end{figure}

\section{Summary}

We have reviewed the role of $N^*(2120)$ resonance in the associate
$K\Lambda(1520)$ photon and hadronic productions at low energies
within an effective Lagrangian approach and the isobar model. In
addition to the contact, $t-$channel $\bar K$ exchange, and
$s-$channel nucleon pole contributions, the contributions from the
$u-$channel $\Lambda(1115)$ hyperon pole term and $N^* (2120)$
resonance are also considered. The results show that when the
contributions from the $N^*(2120)$ resonance and the $\Lambda(1115)$
are taken into account, both the new CLAS~\cite{Moriya:2013hwg} and
the previous LEPS data~\cite{leps1,leps2} for the $\gamma p \to
K^+\Lambda(1520)$ reaction can be simultaneously described.

Actually, we find an overall good description of the data, both at
forward and backward $K^+$ angles, and for the whole range of
measured $\gamma p$ invariant masses. The contribution of the
$u-$channel $\Lambda (1115)$ pole term produces an enhancement at
backward angles, and it becomes more and more relevant as the photon
energy increases, becoming essential above $W \ge 2.35$ GeV and
$\cos\theta_{\rm c.m.} \le -0.5$. On the other hand, the CLAS data,
clearly support the existence of a spin-parity $J^P = 3/2^-$ nucleon
resonance with a mass around 2.1 GeV, a width of at least 200 MeV
and a large partial decay width into $\Lambda(1520)K$. These
characteristics could be easily accommodated within the constituent
quark model results of Simon Capstick and W. Roberts of
Ref.~\cite{simonprd58}. Such resonance might be identified with the
two stars PDG $N^*(2120)$ state, which would confirm previous
claims~\cite{xiejuan,hejun} from the analysis of the bump structure
in the LEPS differential cross section at forward $K^+$ angles
discussed in Fig.~\ref{dcs-leps}.

On the other hand, motivated by the study of the $\gamma p \to
K^+\Lambda(1520)$ reaction, the role of $N^*(2120)$ has also been
investigated in the $\pi^- p \to K^0 \Lambda(1520)$ and $pp \to
pK^+\Lambda(1520)$ reactions. The results show that the contribution
from the $u-$channel $\Sigma^+$ exchange is dominant in the very
near threshold region, but, when the beam energy increases, the
contributions from $s-$channel nucleon pole and $N^*(2120)$
resonance turn to be very important. Furthermore, the invariant mass
distribution and the Dalitz Plot are also predicted which can be
tested by the future experiments.

\section*{Acknowledgments}

This work is partly supported by DGI and FEDER funds, under
contracts FIS2011-28853-C02-01 and FIS2011-28853-C02-02 the Spanish
Ingenio-Consolider 2010 Program CPAN (CSD2007-00042), Generalitat
Valenciana under contract PROMETEO/2009/0090 and by the National
Natural Science Foundation of China under grants 11105126 and
10905046. We acknowledge the support of the European
Community-Research Infrastructure Integrating Activity "Study of
Strongly Interacting Matter" (acronym HadronPhysics2, Grant
Agreement n. 227431) under the Seventh Framework Programme of EU.
Work supported in part by DFG (SFB/TR 16, "Subnuclear Structure of
Matter").