\section{Introduction}
The biomedical scientific articles hold the valuable knowledge of biomedical entities (such as protein, drug, gene) and their relationships. However, with the exponential increase in the volume of biomedical articles \cite{lu2011pubmed}, it is imperative to advance the development of an accurate biomedical text mining tool to extract and curate meaningful information from huge unstructured texts automatically.\\
\indent One of the cardinal tasks in biomedical document processing is Protein-protein interaction (PPI), where the relation (`\textit{interaction}' or `\textit{non-interaction}') between  two protein mentions is identified from the given biomedical text. The knowledge about protein interactions is critical in understanding the biological processes, such as signaling cascades, translations and metabolism, that are regulated by the interactions of proteins that alter proteins to modulate their stability \cite{elangovan2020assigning}.

Majority of the existing works on PPI in the literature primarily focused only on the textual information present in the biomedical article. However, these approaches lack in capturing \textbf{(1)} multi-omnics biological information regarding protein interactions, and \textbf{(2)} genetic and structure information of the proteins. A few works \cite{dutta-saha-2020-amalgamation,asada-etal-2018-enhancing, DBLP:conf/iconip/JhaSK20, Jha2020} have been reported in the literature where the researchers have considered different modalities of the biomedical corpus.
 However, these multi-modal architectures are modality-specific and thus are very complex. Hence, there is a surge to develop a generalized and optimized model that can understand all the modalities rather than developing various architectures for different modalities.  \\
\indent Towards this, we explore Graph-based Transformer model (GraphBERT) \cite{zhang2020graph} to learn the modality independent graph representation. This enables the model to acquire the joint knowledge of both the modalities (textual and protein structure) under a single learning network.
The main contributions of this work are:
\begin{enumerate}[noitemsep]
    \item Besides the textual information of the biomedical corpus, we have also utilized protein atomic structural information while identifying the protein interactions.
    \item Developed a generalized modality-agnostic approach that is able to learn the feature representations of both the textual and the protein-structural modality.
    \item Our analysis reveals that addition of protein-structure modality increases the efficiency of model in identifying the interacted protein mentions.
\end{enumerate}
\paragraph{Related Work:}\label{related}
Existing studies have adopted traditional statistical and graphical methods \cite{miyao2008evaluating,chang2016pipe} to identify the protein interactions from the textual content.
Later, with the success of deep learning, several techniques based on Convolutional Neural Network \cite{choi2018extraction,peng2017deep,ekbal2016deep}, Recurrent Neural Network \cite{hsieh2017identifying,ahmed2019identifying}, Long Short Term Memory network \cite{yadav2019feature,ningthoujam2019relation,yadav2020relation}, and language models \cite{yadav2021they} based methods are proposed for extracting the relationships from biomedical literature and clinical records.
\citet{fei2020span} proposed a span-graph neural model for jointly extracting overlapping entity relationships from biomedical text.
The recent advancement of the Transformer model \cite{lee2020biobert,beltagy2019scibert} in the biomedical domain has also led to significant performance improvement in biomedical relation extraction task \cite{giles2020optimising}.
Recently, the use of multi-modal dataset in BioNLP domain \cite{dutta-saha-2020-amalgamation, asada-etal-2018-enhancing} draws the attention of the researchers due to its better performance than the traditional approaches. In contrast, our model is independent of handling multiple modalities without relying on modality-specific architectures.

\section{Proposed Method}\label{proposed}
\begin{figure*}
\centering
\includegraphics[width=\textwidth]{proposed_methodology.png}
\caption{An outline of the proposed deep multi-modal architecture for PPI.}
\label{figure-1}
\end{figure*}

In this section, we introduce our proposed method and its detailed implementation.
The proposed deep multi-modal architecture is illustrated in Figure-\ref{figure-1}, that consists of four main components:
\textbf{(1)} \textit{Multi-modal Graph Constructor}, \textbf{(2)} \textit{Multi-modal Graph Fusion}, \textbf{(3)} \textit{Multi-modal Graph Encoder}, \textbf{(4)} \textit{PPI Predictor}.
Below we briefly describe each of the model components.
\paragraph{Problem Statement:} Given a biomedical input text $S=\{w_1, w_2, \ldots, w_n\}$ having $n$ words, and a pair of protein mentions $p_1, p_2 \in S$, we aim to predict, whether the protein mentions will \textit{`interact'} or \textit{`non-interact'}.
\subsection{Multi-modal Graph Constructor} This component consists of two distinct graph constructors for two different modalities, which are Textual Graph Constructor and Protein Structure Graph Constructor. The former, constructs the graph by considering the textual content that aims to capture the lexical and contextual information present in the input. The later, exploits the atomic structure (3D PDB structure) of the protein molecules to build the graph.
\paragraph{Textual Graph Constructor:}
To generate the textual graph, we begin by first constructing the vocabulary from the training corpus. For each input text $S$, we use one-hot-encoding mechanism to encode them as a vector representation $R_S \in \mathcal{R}^{|V|}$. However, the representation $R_S$ suffers from the data sparsity as the vocabulary size can become very large for the entire training corpus. To deal with this, we utilized the Principal Component Analysis (PCA) \cite{wold1987principal} to reduce the vector dimensionality. The textual graph $\mathcal{G}_T =\{\mathcal{V}_T, \mathcal{E}_T \}$ is formulated by the nodes $\mathcal{V}_T=\{ \hat{R}_{S_1}, \hat{R}_{S_2}, \ldots, \hat{R}_{S_{|N|}}\}$, where $|N|$ is the number of input sentences in the training dataset and $\hat{R}_{S_i} \in \mathcal{R}^{|\hat{V}|}$ is reduced vector representation of size $|\hat{V}|$ for sentence $S_i$. The link $e_{i,j}$ between nodes $\hat{R}_{S_i}$ and $\hat{R}_{S_j}$ is determined by the common entities (\textit{protein}) present in both the sentences $S_i$ and $S_j$, if there is no common entity, then link does not exist between the nodes. The edges $\mathcal{E}_T = \{ e_{i, j} \ | \ i, j \in \mathcal{V}_T, \ \text{and} \ \text{\textit{protein}} \in i, \ j \}$ are the set of all the links that exist between any two nodes in the graph, $\mathcal{G}_T$.

\paragraph{Protein Structure Graph Constructor:}
For the protein structural modality, we created a graph where each node represents an atom and the edge represents the connection between the atoms. To obtain the atomic information about the proteins, first we have mapped the proteins into genes and utilized the PDB (Protein Data Bank)\footnote{https://www.rcsb.org/} for each associated protein mention.
Each protein information obtained from PDB consists of set of atoms $\{a_1, a_2, \ldots,  a_A\}$, and a
node feature matrix, $N_p \in \mathcal{R}^{A \times d_p}$. The node feature matrix for each protein $k$ undergoes the convolutional operation \textit{CNN(.)} followed by the max-pooling operation, $pool(.)$. Formally, $P_{k}=pool(relu(\text{\textit{CNN}}(N_{p_k})))$. The final protein representation, $P_{S_i}$, for both the proteins present in the given input sentence $S_i$ is computed as follows: $P_{S_i} = P_1 \oplus P_2$. Following this, the protein structure graph $\mathcal{G}_P =\{\mathcal{V}_P, \mathcal{E}_P \}$ is formulated by the nodes $\mathcal{V}_P=\{ {P}_{S_1}, {P}_{S_2}, \ldots, {P}_{S_{|N|}}\}$, where $|N|$ is the number of input sentences in the training dataset and ${P}_{S_i} \in \mathcal{R}^{d_s}$ is the protein structure representation of size $d_s$ for sentence $S_i$.

\subsection{Multi-modal Graph Fusion}
In this component, we fused the textual graph $\mathcal{G}_T$ and protein structure graph $\mathcal{G}_P$ with the aim of generating a joint representation that is capable of capturing the contextual, lexical, and multi-omnics information. Towards this, we expanded the node information of textual graph with the node information obtained in the protein-structure graph. Specifically, we created a multi-modal graph $\mathcal{G}$ with the nodes $\mathcal{V}$ having concatenated vector representations from the respective nodes of textual graph and protein structure graph. Formally,
\begin{equation}
    \mathcal{V}_i = \hat{R}_{S_i} \oplus {P}_{S_i}
\end{equation}
The link information remains intact in the multi-modal graph fusion, thus, $\mathcal{E} = \mathcal{E_T}$.

\subsection{Multi-modal Graph Encoder} Majority of the existing works on multi-modal relation extraction have treated multiple modalities separately and exploited the modality-specific architectures to learn the respective feature representations. However, these strategies inhibit the learning of inherent shared complementary features, that are often present across the modalities.
To address this, we present an end-to-end multi-modality learning mechanism that exploits the single expanded multi-modal graph (obtained from the Multi-modal Graph Expansion component) with the Graph-based Transformer encoder. Specifically, we utilized the Graph-BERT \cite{zhang2020graph} encoder over the other dominants graph neural networks (GNNs) primarily due to its capability to avoid the \textbf{(a)} \textit{suspended animation problem} \cite{zhang2019gresnet}, and \textbf{(b)} \textit{over-smoothing problem} \cite{li2018deeper} that hinders the applications of GNNs for deep graph representation learning tasks.
For a given multi-modal graph $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ with the set of nodes ($\mathcal{V}$) and edges ($\mathcal{E}$), Graph-BERT sampled set of graph batches for all the nodes as set $\mathcal{G} = \{g_1, g_2, \ldots, g_{|\mathcal{V}|}\}$. For all the nodes $v_j$ in sub-graph $g_i$, the Graph-BERT computes raw feature vector embedding $e_j^{x}$, role embedding $e_j^{r}$, position embedding $e_j^{p}$ and distance embedding $e_j^{d}$. The initial input vector for node $v_j$ is computed as follows: $h_{j}^{(0)}= e_j^{x} + e_j^{r} + e_j^{p} +e_j^{d}$. Furthermore, the initial input vectors for all the nodes in $g_i$ can be organized into a matrix $H^{(0)} =
[h_i^{(0)}, h_{i,1}^{(0)}, \ldots, h_{i,k}^{(0)}]^\top$, where $k$ is a hyper-parameter. The Graph-Transformer \cite{zhang2020graph} computes the vector representation of $D$ layers of transformers. The final feature ($z_i$) for node $v_j$ is computed as follows:
\begin{equation}
\begin{aligned}
{H}^{(0)} &  = [{h}_i^{(0)}, {h}_{i,1}^{(0)}, \cdots, {h}_{i,k}^{(0)}]^\top \\
{H}^{(l)} &= \text{\textit{G-Transformer}} \left( {H}^{(l-1)}\right)\\
{z}_i &= \sum_{m=0}^{m=D} {H}^{(m)}
\end{aligned}
\end{equation}

\subsection{PPI Predictor}
The final feature ($z_i$) of each node $i$ is used to predict  the PPI category. Towards this, we employed a feed-forward network with \textit{softmax} activation layer to predict the input text into one of the two classes \textit{interaction} or \textit{non-interaction}. Formally,
\begin{equation}
\footnotesize
    prob ({c=interact}|\mathcal{G}, S, \theta) = \frac{exp(\bm{W}^{T}z_i^{\text{interact}}+\bm{b})}{\sum_{k=1}^{K}{exp(\bm{W}^{T}z_i^{k}+\bm{b})}}
\end{equation}
where, $\bm{W}$ and $\bm{b}$ are the weight matrix and bias vector, respectively. $K$ denotes total number of distinct classes, which are \textit{`interaction'} and \textit{`non-interaction'} in our case.

\begin{table}[t]
\begin{adjustbox}{max width=\linewidth}
\begin{tabular}{@{}lllcccl@{}}
\toprule \toprule
 &  &  & Precision & Recall & F-score &  \\ \midrule
 & \textbf{Proposed Model} &  & 80.84 & {80.87} & 80.86 &  \\
 & \citet{dutta-saha-2020-amalgamation} &  & 69.04 & 88.49 & 77.54 &  \\
 & \citet{yadav2019feature} &  & 80.81 & 82.57 & 81.68 &  \\
 & \citet{hua2016shortest} &  & 73.40 & 77.00 & 75.20 &  \\
 & \citet{choi2018extraction} &  & 72.05 & 77.51 & 74.68 &  \\
 & \citet{qian2012tree}  & & 63.61 & 61.24 & 62.40 &  \\
 &  \citet{peng2017deep} &  & 62.70 & 68.2 & 65.30 &  \\
 & \citet{zhao2016protein} &  & 53.90 & 72.9 & 61.60 &  \\
 & \citet{tikk2010comprehensive} &  & 53.30 & 70.10 & 60.00 &  \\
 & \citet{li2015approach} &  & 72.33 & 74.94 & 73.61 &  \\
 & \citet{choi2010simplicity} &  & 74.50 & 70.90 & 72.60 &  \\
 \bottomrule \bottomrule
\end{tabular}
\end{adjustbox}
\caption{Comparative analysis of the proposed multi-modal approach with state-of-the-art techniques for BioInfer dataset.}\label{table:2}
\vspace{-2mm}
\end{table}
\vspace{-2mm}
\begin{table}[t]
\begin{adjustbox}{max width=\linewidth}
\begin{tabular}{@{}clccc@{}}
\toprule
 &  & Precision & Recall & F-score \\ \midrule
\multirow{2}{*}{HPRD50} & Textual Modality & 90.44 & 92.18 & 91.28 \\
 & Proposed Model & 95.47 & 94.69 & 95.06 \\ \midrule
\multirow{2}{*}{BioInfer} & Textual Modality & 78.49 & 79.78 & 79.06 \\
 & Proposed Model & 80.84 & 80.87 & 80.86 \\ \bottomrule
\end{tabular}
\end{adjustbox}
\caption{Results by uni-modal and multimodal approaches}\label{table:1}
\end{table}
\section{Datasets and Experimental Analysis}\label{result}
{\bf Datasets:}
In this work, we have collected two exemplified multi-modal protein protein interaction datasets \cite{dutta-saha-2020-amalgamation}. In these datasets, the authors exemplified two popular benchmark PPI corpora, namely BioInfer\footnote{http://corpora.informatik.hu-berlin.de/} and HPRD50\footnote{https://goo.gl/M5tEJj}.

\paragraph{Experimental Setup}
We have utilized the pre-trained Graph-BERT\footnote{https://github.com/jwzhanggy/Graph-Bert} in our experiment.
The initial vocabulary for BioInfer and HPRD50 datasets are 6561 and 1277, respectively. We have projected them into 1000 and 1185 dimension vectors using PCA, respectively. We have kept maximum of 5052 and 1185 number of words in both the datasets, respectively. The filter-size of CNN is set to $3,4$. We  have obtained $1185$ length node feature representation for protein structure graph. The nodes of multi-modal graph received the $2185$ sized feature representation. We have obtained $2500$ and $25859$ number of nodes and edges from HPRD50 dataset and 13675 and 15930214 number of nodes and edges from BioInfer dataset for the Graph-BERT training, respectively. We have used all the hyper-parameters of Graph-BERT model in our proposed model. We have kept following hyper-parameters values: subgraph size = 5, hidden size = 32, attention head number = 2, Transformer layers, D = 2, learning rate = 0.01, weight decay = 5e-4, hidden dropout rate = 0.5, attention dropout rate = 0.3, loss = cross entropy, optimizer = adam \cite{adam}. The hyper-parameters are chosen based on the 5-fold cross-validation experiments on both the datasets.

{\bf Results and Analysis:}
We have compared the performance (\textit{c.f.} Table-\ref{table:2},\ref{table:3}) of our proposed model with the existing state-of-the-art methods on PPI for both the datasets. These existing methods are based on different techniques like kernel-based \cite{choi2010simplicity, tikk2010comprehensive, qian2012tree, li2015approach}, deep neural network-based \cite{zhao2016protein, yadav2019feature}, multi-channel dependency-based convolutional neural network model \cite{peng2017deep}, semantic feature embedding \cite{choi2018extraction}, shortest dependency path \cite{hua2016shortest} and a recent deep multi-modal approach \cite{dutta-saha-2020-amalgamation}. It is to be noted that our results on BioInfer and HPRD50 are not directly comparable with the existing approaches as other methods have utilized different test sets for evaluation.
From the above comparative study, it is evident that our proposed multi-modal approach identifies the protein interactions in an efficient way and can be further improved in different ways.

\begin{table}[t]
\begin{adjustbox}{max width=\linewidth}
\begin{tabular}{@{}lllcccl@{}}
\toprule \toprule
&  &  & Precision & Recall & F-score &  \\ \midrule
 & \textbf{Proposed Model} &  & 95.47 & 94.69 & 95.06 &  \\
  & \citet{dutta-saha-2020-amalgamation} &  & 94.79 & 75.21 & 83.87 &  \\
 & \citet{yadav2019feature} &  & 79.92 & 77.58 & 78.73 &  \\
 & \citet{tikk2010comprehensive} &  & 68.20 & 69.80 & 67.80 &  \\
 & \citet{tikk2010comprehensive}(with SVM) &  & 68.20 & 69.80 & 67.80 &  \\
 & \citet{palaga2009extracting} &  & 66.70 & 80.20 & 70.90 &  \\
 & \citet{airola2008all}(APG) &  & 64.30 & 65.80 & 63.40 &  \\
 & \citet{van2008extracting} &  & 60.00 & 51.00 & 55.00 &  \\
 & \citet{miwa2009protein} &  & 68.50 & 76.10 & 70.90 &  \\
 & \citet{airola2008all}(Co-occ) &  & 38.90 & 100 & 55.40 &  \\
 & \citet{pyysalo2008comparative} &  & 76.00 & 64.00 & 69.00 &  \\
 \bottomrule \bottomrule
\end{tabular}
\end{adjustbox}
\caption{Comparative analysis of the proposed multi-modal approach with other state-of-the-art approaches for HPRD50 dataset.}\label{table:3}
\end{table}

\begin{figure*}[!t]
\centering
\includegraphics[width=0.70\textwidth]{picture/analysis.png}
\caption{Confusion matrices of our proposed approach on both HPRD and BioInfer datasets with only textual modality and text+structure modality.}\label{figure-2}
\end{figure*}
\noindent \textbf{Discussion:}
To analyze the role of each modality, we conducted ablation study as shown in Table \ref{table:1}.  We performed the experiments with the textual modality. Here, we could not consider the protein-structural modality alone as it would bring the conflicting labeling relation. For example, consider two sentences that contain same pair of proteins but these proteins can have conflicting relations (interacting or non-interacting) depending on the context of sentences in which they appear. Hence, we could not consider the protein-structural modality alone. Though the structural modality is unable to draw any conclusion alone, however the integration of both the modalities demonstrates the improvements (3.78\% and 1.8\%, in terms of F-score for HPRD50 and BioInfer, respectively) over the textual modality alone.

\section{Error Analysis}
The comparative confusion matrices with only textual-modality and multi-modality for both the datasets are shown in Figure-\ref{figure-2}. We have performed error analysis to postulate possible reasons and areas with scope of improvement in our experiments. After careful study on false positive and false negative classes, following observations can be made.\\1) Instances with a large number of protein mentions in a single sentence can cause misclassification. For example, the maximum number of proteins in any instances of BioInfer and HPRD50 datasets are 26 and 24, respectively. These large number of proteins present in a single instance may lead the network to misclassificaton.\\
    2) Few samples contain repeated mentions of the same protein. This adds noise and might lead to losing useful contextual information.\\
   3) To get a consistent graph from molecular structure, the nodes were required to be of the same length. This is done by padding the vectors with zeros, and when the PDB is not available, a null vector is used for consistency. A better handling of missing data will help in learning the proposed model.

\section{Conclusion}\label{conclusion}
This work presents a novel modality-agnostic Graph-based framework to identify the interactions between the proteins. Specifically, we explored two modalities: \textit{textual}, and \textit{molecular structure} that enable the model to learn the domain-specific multi-omnics information complementary with the task-specific contextual information.
A detailed comparative results and analysis proves that our proposed multi-modal approach can capture underlying molecular structure information without relying on sophisticated modality-specific architectures.
Future work aims at extending this study to the other related tasks like drug-drug interactions.
\section*{Acknowledgment}
Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (Formerly Media Lab Asia) for carrying out this research.

\bibliographystyle{acl_natbib}


\section{Introduction}

E-commerce search, which serves as an essential part of online shopping platforms, typically consists of three steps: query processing, candidate retrieval and ranking. In this paper, we focus solely on the candidate retrieval step which aims to retrieve thousands of candidates from billions of items.

Despite the recent advance of embedding based retrieval methods~\cite{zhang2020towards} and their successful applications in candidate retrieval stage, in practice we still notice a few drawbacks: 1) representations of low-frequency items are not well learned due to their rare occurrences; 2) Query embeddings are computed purely from a handful of query tokens, which sometimes can not capture the query semantic meanings well; 3) The training efficiency is low, mainly because that each training example consists of only one pair of query and clicked item.

In this paper, we explore Graph Convolution Networks (GCN) to enrich entity representations by aggregating related neighbor's information. In our case of e-commerce search, it is very likely that we can learn a better embedding representation for a user query, if we can aggregate information from the most clicked or bought items for the query to alleviate the shortage of query tokens. Likewise, we can learn a better item embedding representation, if we can aggregate information from the most frequent queries that lead to the item clicks or orders. Figure~\ref{fig:ecomm_graph} shows a quick glance of typical e-commerce graph  which illustrates how a query aggregates information from related items. Moreover, we explore how to
improve the training efficiency by neighbor sampling.

\begin{figure}[!t]
    \centering
    \hspace*{4mm}
    \includegraphics[width=0.45\textwidth]{graph/graph_figure.pdf}
    \caption{Illustration of e-commerce search graph.}
    \label{fig:ecomm_graph}
    \vspace{-6mm}
\end{figure}

\section{Related Work}
\label{sec:related_work}

\subsection{Graph Convolution Network}
Neural networks that operate on graphs have previously been introduced in~\cite{gori2005} and~\cite{scarselli2009} as a form of recurrent neural network. Their frameworks apply propagation functions until node representations reach a stable fixed point. Later on, with the success of convolution neural networks in various domains, such as image classification~\cite{he2016}, semantic segmentation~\cite{jegou2017}, text classification~\cite{kim2014}, and machine translation~\cite{gehring2017},
there is an increasing interest in generalizing convolution network to graph domain. Advances in this Graph Convolution Network (GCN) directions are often categorized into spectral approaches and non-spectral approaches.

Spectral approaches work with a spectral representation of the graph. Originally, in the work of \citeauthor{bruna2013spectral}, the convolution operation is defined in the Fourier domain by computing the eigendecomposition of the graph Laplacian, which, however, requires intense computations.
\citeauthor{kipf2016semi} alleviated the computational cost by developing an efficient layer-wise propagation rule based on a first-order approximation of spectral graph convolutions.
However, the layer-wise propagation rule still have two drawbacks during training procedure. First, the graph convolution requires full graph Laplacian, which is computational-expensive and even problematic for training with large graph. Second, the full graph Laplacian is computed from a fixed graph, which do not naturally generalize to unseen nodes.

The other non-spectral approaches define convolutions directly on the graph, by operating on a set of spatially close neighbors. GraphSAGE~\cite{hamilton2017inductive} avoided operating on the entire graph Laplacian by developing a learnable aggregation functions, which comprehensively scale the GCN algorithm to large graph and achieve the ability to generate node embeddings for previously unseen data. Additionally, the neighbor sampling in the work of GraphSAGE alleviate the computation cost to receptive fixed field during mini-batch training procedure. Later on, PinSAGE \cite{ying2018graph} scaled the GCN algorithm to the real item-item recommendation task with large graph having billions of nodes and edges, which paves the way of large-scale deep recommendation systems based on GCN architectures.

Recently, there are plenty of GCN variants proved to be more effective than previous first wave of GCN exploration. LightGCN~\cite{he2020lightgcn} simplifies the GCN algorithm significantly by reducing the unnecessary and even noisy operations, feature transformation and nonlinear activation, and achieves state-of-the-art results on recommendation tasks.
Graph Attention Network \cite{velivckovic2017graph} leverages attention mechanisms in the neighbor aggregation process to improve the performance.

Our work differs from the above approaches, in that we develop a simple and effective GCN solution for a web-scale e-commerce embedding retrieval problem. To the best of our knowledge, we are the first to explore GCN's application to search embedding retrieval task. Accordingly, to make this happen, we have tried many techniques. In this paper, we will share our experience and present the best known approach.

\subsection{Embedding Retrieval in Search Engine}
Recently, embedding retrieval technologies have been widely adopted in the modern recommendation and advertising systems~\cite{youtube2016, mind2019, tree2018}, while have not been widely used in search engine yet. We find a few works about retrieval problems in search engine~\cite{palangi2016deep,vu2017search}, while they have not been applied to the industrial production system. DPSR~\cite{zhang2020towards} is one of the first practical explorations in this direction of applying embedding retrieval in the industrial search engine system. Moreover, the well known DSSM~\cite{Huang:2013:LDS:2541176.2505665} and its following work  CDSSM~\cite{Shen:2014:LSR:2567948.2577348} have pioneered the work of using deep neural networks for relevance scoring, which is very different task from embedding retrieval though.

\section{Method}
\label{sec:method}

Given a graph $\G = (\mathcal{V}, \mathcal{E})$ where each $v\in \mathcal{V}$ stands for an entity node, and each $e \in \mathcal{E}$ stands for an edge between two entities, let's denote the set of all the neighbors for a node $v$ as $\mathcal{N}_v$, the set of queries as $\mathcal{Q}$, and the set of items as $\mathcal{I}$, where $\mathcal{Q} \subseteq \mathcal{V}$ and $\mathcal{I} \subseteq \mathcal{V}$.

\header{Neighbor Aggregation.}
The core of GCN is to compute the $l$-th layer's representation of a target node by all its neighbors and itself in $(l-1)$-th layer. Formally, we can define the aggregation operation as follows
\begin{equation*}
\mathbf{h}_{v}^{l} = \text{AGGREGATE}(\mathbf{h}_v^{l-1}, \mathbf{h}_{u}^{l-1}, \forall u \in \mathcal{N}_{v} ),
\end{equation*}
where $\mathbf{h}_v^l\in \mathbb{R}^{d}$ denotes the $d$-dimensional embedding representation of a node $v$ at $l$-th layer, the AGGREGATE function is a pooling function that takes the target node and all its neighbor nodes as inputs.

In SearchGCN, we use the following neighbor aggregation function that can be explained as a weighted sum of all its neighbors,
\begin{equation}
\mathbf{h}_{v}^{l} = \sum_{u \in \mathcal{N}_{v}} \alpha^{l-1}_{vu} \mathbf{h}_{u}^{l-1},
\label{eq:attention_aggregation}
\end{equation}
\begin{equation}
\alpha^{l-1}_{vu} = \frac{\text{exp}\left(\text{LeakyReLU}\left({\mathbf{w}^{l-1}}^\top [\mathbf{h}^{l-1}_{v}||\mathbf{h}^{l-1}_{u}]\right) \right)}
{\sum_{u\in\mathcal{N}_{v}} \text{exp} \left( \text{LeakyReLU}\left({\mathbf{w}^{l-1}}^\top [\mathbf{h}^{l-1}_{v}||\mathbf{h}^{l-1}_{u}]\right) \right)},
\label{eq:attention_weight}
\end{equation}
where $.^\top$ denotes the transpose operation, $||$ denotes the concatenation operation and $\mathbf{w}^{l-1} \in \mathbb{R}^{2d}$ is model parameter to learn.

\header{Layer Combination.}
In our {SearchGCN}, to refine the embedding of query and item node, we sum the neighbor embeddings obtained by neighbor aggregation layer-by-layer to form the final embedding:
\begin{equation}
\mathbf{z}_{v} = \sum_{l=0}^{L} \mathbf{h}_{v}^{l}. \nonumber
\end{equation}

\header{Neighbor Sampling.}
Since our e-commerce graph has hundreds of millions of nodes and billions of edges, which is too large to be fully loaded into computer memory, and the e-commerce graph appears to be significant imbalance,  which makes the neighbor aggregation expensive. Therefore, to be practical in an industrial system, we need to develop a neighbor sampling approach that is efficient in both memory and computation.

First, we prune the neighborhood size for each node to at most $50$, by the weights of the edges between each neighbor and the target node, to allow us to load all edges into memory during the training. Second, we introduce randomness to neighbor sampling, by sampling some fixed number (\textit{e.g.}, $10$) of neighbors according to edge weights, to reduce the computational cost to aggregate neighbors and increase model generalization ability.

\section{Experiments}
\label{sec:experiment}
We conduct experiments on a data set of $60$ days' user click logs and we can assume that if an item is clicked for a query, the item is relevant to the query.

\header{Offline Evaluation.}
We compare {SearchGCN} with a few baseline methods and report the evaluation results in Table~\ref{tab:offline_comparison}. \textit{SearchGCN-mean} adopts mean aggregator that gives equal weight to all neighbors. \textit{SearchGCN-attention} adopts attention mechanism in aggregation function as shown in Eqs. (\ref{eq:attention_aggregation}) and (\ref{eq:attention_weight}). \textit{SearchGCN-mask} adopts neighbor masking on both query and item side to mask the target query and item.

\begin{table}[tb]
    \centering
    \caption{Comparison between different methods.}
    \label{tab:offline_comparison}
    \begin{tabular}{c|cccc}
    \hline
        & Top-$1$   & Top-$10$   & AUC   & Sec./Step \\
    \hline
    DPSR~\cite{zhang2020towards} & $86.23\%$ & $99.59\%$ & $0.737$ & $0.276$ \\
    GraphSAGE~\cite{hamilton2017inductive} & $79.29\%$ & $95.97\%$ & $\mathbf{0.783}$ & $1.03$ \\
    PinSAGE~\cite{ying2018graph} & $68.59\%$ & $93.18\%$ & $0.736$ & $1.07$ \\

    \hline
    {SearchGCN}-mean & $86.76\%$ & $99.43\%$ & $0.764$ & $1.04$ \\
    {SearchGCN}-attention & $86.57\%$ & $99.27\%$ & $0.751$ & $1.05$ \\
    {SearchGCN}-mask & $\mathbf{87.72\%}$ & $\mathbf{99.67\%}$ & $0.768$ & $0.98$ \\
    \hline
    \end{tabular}
\end{table}

\header{Online A/B Tests.} We conducted live experiments over $10\%$ of the entire site traffic during a period of one week using a standard A/B testing configuration. To protect confidential business information, only relative improvements are reported. Table~\ref{tab:abtest} shows that the proposed {SearchGCN} retrieval improves the production system for core business metrics in e-commerce search, including user conversation rate (UCVR), revenue per mille (RPM) and gross merchandise value (GMV). As a result, the proposed {SearchGCN} has been deployed into JD.com's search production since July 2020.

\begin{table}[t]
\centering
    \caption{Improvements of {SearchGCN} in online A/B test.}
    \label{tab:abtest}

    \begin{tabular}{c|r r r r}
    \hline
            & UCVR  & RPM & GMV  \\
    \hline
    SearchGCN  & $+0.20\%$ & $+1.13\%$ & $+1.27\%$  \\
    \hline
    \end{tabular}

\end{table}

\header{Improvement on Torso and Tail Query.}
We evaluate {SearchGCN}'s performance on different types of queries. We split our evaluation data set into three categories, head, torso and tail queries, according to the queries' frequency.
As shown in Table~\ref{tab:tail_query}, we can see that the improvements on torso and tail queries are more significant than head queries. This observation strongly suggests that the GCN's neighbor aggregation could benefit more for the torso and tail queries, potentially because they can learn from their neighbor's representations. Similar trend also holds for long tail items, but we have to skip due to space limit.

\begin{table}[t]
    \centering
    \caption{Comparison between different methods on head, torso, and tail queries. The metric is top-1 score.}
    \label{tab:tail_query}
    \begin{tabular}{c|ccc}
    \hline
      & Head & Torso  & Tail \\
    \hline
    DPSR & $84.23\%$ & $92.61\%$ & $92.64\%$ \\
    SearchGCN & $85.73\%$ & $94.68\%$ & $94.77\%$ \\
    \hline
    Difference & $1.50\%$ & $2.07\%$ & $2.13\%$ \\
    Error reduction rate & $9.51\%$ & $28.01\%$ & $28.94\%$ \\
    \hline
    \end{tabular}
    \vspace{-4mm}
\end{table}
\section{Conclusion}
\label{sec:conclusion}
In this paper, we have proposed a GCN based approach for embedding retrieval in search engine. We also extensively study the offline comparison with other baseline models, and the results show that the proposed SearchGCN is both effective in retrieval quality metrics and training efficiency. Consequentially, we have successfully deployed SearchGCN to one of largest e-commerce search engine in the world, with significant business metric gains in the online A/B test.

\section{Company Profile}

\section{Presenter Profiles}
\header{Xinlin Xia} is a researcher in the Department of Search and Recommendation at JD.com Beijing. He received his master degree in Computer Science from Sichuan University, China. His research focuses on information retrieval.

\header{Han Zhang} is a researcher in the Department of Search and Recommendation at JD.com Beijing. She graduated from Natural Language Processing Lab, Department of Computer Science and Technology, Tsinghua University. Her research focuses on information retrieval.

\header{Wen-Yun Yang} is a principal scientist at JD.com and previously a research scientist at Facebook News Feed team. He had about 8 years experience of tech leading in search and recommendation backend systems, candidate retrieval, ranking and modeling in general. He received his Ph.D. degree in Computer Science from University of California, Los Angeles.

\end{document}


\section{INTRODUCTION}
\label{intro}

\begin{figure}[t]
    \includegraphics[width=\columnwidth]{hart_architecture.png}
    \caption{Overview of the Action Transformer architecture. Pose estimations are linearly projected to the dimension of the model, and together with the class token, they form the input tokens of the transformer encoder. As for Vision Transformer models \cite{dosovitskiy2020image, touvron2020training,pmlr-v139-d-ascoli21a}, a learnable positional embedding is added to each input token. Then, only the output class token is passed through a multi-layer perceptron head to obtain the final class prediction.}
    \label{fig:act_arch}
\end{figure}

Human Action Recognition (HAR) is a problem in computer vision and pattern recognition that aims to detect and classify human actions. The ability to recognize people inside a scene and predict their behavior is fundamental for several applications, such as robotics \cite{rodriguez2020shedding}, surveillance and security \cite{Vallathan2021, xuanming2021}, autonomous vehicles \cite{martin2019drive, ben2021driving}, and automatic video captioning \cite{zhou2018end, tu2021enhancing, wan2022revisiting}. Most previous works dealing with HAR adopted datasets characterized by samples with a long temporal duration \cite{HUANG2019165, 7940083, luvizon2020multi}. Thus, HAR has been mainly treated as a post-processing operation, classifying complex and long-lasting human actions by exploiting past and future information. Conversely, in this work, we focus on short-time HAR, which aims at continuously classifying actions within short past time steps (up to a second). This approach is fundamental to target real-time applications: in robotics, for example, HAR problem should be solved promptly to react to sudden behavioral changes, only relying on near past information.

In this paper, we propose a new model for HAR called the Action Transformer (AcT), schematized in Fig.\ref{fig:act_arch}, inspired by the simple and prior-free architecture of the Vision Transformer \cite{dosovitskiy2020image}. The Transformer architecture \cite{vaswani2017attention} has been one of the most important deep learning advances of the last years in natural language processing (NLP). Moreover, multi-head self-attention has proven to be effective for a wide range of tasks besides NLP, e.g. image classification \cite{dosovitskiy2020image,touvron2020training,pmlr-v139-d-ascoli21a}, image super-resolution \cite{Yang_2020_CVPR, zhu2020learning}, and speech recognition \cite{8462506}. Furthermore, optimized versions of the Transformer have been developed for real-time and embedded applications \cite{berg2021keyword}, proving that this architecture is also suitable for Edge AI purposes.
Recently, many models for Human Action Recognition have proposed the integration of attention mechanisms with convolutional and recurrent blocks to improve the accuracy of models. However, solutions that rely exclusively on self-attention blocks have not been investigated for this task yet.

With AcT, we apply a pure Transformer encoder derived architecture to action recognition obtaining an accurate and low-latency model for real-time applications. We study the model at different scales to investigate the impact of the number of parameters and attention heads. We use the new single-person, short-time action recognition dataset MPOSE2021 as a benchmark and exploit 2D human pose representations provided by two existing detectors: OpenPose \cite{cao2019openpose} and PoseNet \cite{papandreou2018personlab}. Moreover, we compare AcT with other state-of-the-art baselines to highlight the advantages of the proposed approach. To highlight the effectiveness of self-attention, we conduct a model introspection providing visual insights of the results and study how a reduction of input temporal sequence length affects accuracy. We also conduct extensive experimentation on model latency on low-power devices to verify the suitability of AcT for real-time applications.

The main contributions of this work can be summarized as follows:
\begin{itemize}
    \item We study the application of the Transformer encoder to 2D pose-based HAR and propose the novel AcT model, proving that fully self-attentional architectures can outperform existing convolutional and recurrent models for pose-based HAR.
    \item We introduce MPOSE2021, a dataset for real-time short-time HAR, suitable for both pose-based and RGB-based methodologies. It includes 15429 sequences from 100 actors and different scenarios, with limited frames per scene (between 20 and 30). In contrast to other publicly available datasets, the peculiarity of having a constrained number of time steps stimulates the development of real-time methodologies that perform HAR with low latency and high throughput.
\end{itemize}

The rest of the paper is organized as follows. Section \ref{relatedworks} briefly reports the relevant research on HAR and self-attention deep learning methodologies. Section \ref{mpose2021} presents the MPOSE2021 dataset, highlighting features and elements of novelty with respect to other existing datasets. Then, Section \ref{actiontransformer} describes the architecture of the proposed AcT model, shortly recalling the main features of the Transformer encoder. Section \ref{experiments} summarizes the experimentation conducted to verify the effectiveness of the proposed approach, gives a visual insight into the model functioning, studies its behavior under temporal information reduction, and measures its latency for real-time applications. Section \ref{conclusion} draws some conclusions for this work and indicates future research directions.

\section{RELATED WORKS}
\label{relatedworks}
\begin{figure}[b]
    \centering
    \includegraphics[width=\linewidth]{mpose2021_summary.pdf}
    \caption{The number of samples of MPOSE2021 divided by action. The colors show the distribution of precursor datasets among classes, highlighting the unbalanced nature of the data. The final dataset contains 15429 samples where each sample represents one of 100 actors performing one of 20 actions.}
    \label{fig:mpose2021-summary}
\end{figure}
Human Action Recognition algorithms aim at detecting and classifying human behaviors based on a different source of information. Works in this field can be mainly subdivided into two broad categories: video-based and depth-based methodologies\cite{song2021human}. In both, the input consists of a sequence of points representing body joints, extracted from the corresponding RGB frame\cite{LI2020107037}, in the case of video-based methods, or a point cloud, in the case of depth-based ones. In the latter, body joints are provided as 3D coordinates (skeletal data), such as those captured by Kinect sensors \cite{shahroudy2016ntu,liu2019ntu}, that can be possibly projected onto the 2D space \cite{kay2017kinetics,yan2018spatial}. On the opposite, in this work, we focus on a video-based analysis, where body joints are already provided as 2D coordinates (pose data) by pose detection algorithms such as OpenPose \cite{cao2019openpose} and PoseNet \cite{papandreou2018personlab}. This characteristic makes 2D HAR methodologies applicable in a great range of applications using a simple RGB camera. Conversely, skeletal data require particular sensors to be acquired, such as Kinect or other stereo-cameras. That raises substantial limitations, such as availability, cost, limited working range (up to 5-6 meters in the case of Kinect \cite{langmann2012depth}), and performance degradation in outdoor environments.

Angelini et al.\ first collected the MPOSE dataset for video-based short-time HAR \cite{angelini2018actionxpose}, obtaining 2D poses by processing several popular HAR video datasets with OpenPose. Moreover, the authors proposed ActionXPose, a methodology that increases model robustness to occlusions. The sequences were classified using MLSTM-FCN \cite{karim2018multi}, which exploits a combination of 1D convolutions, LSTM \cite{hochreiter1997long}, and Squeeze-and-excitation attention \cite{hu2018squeeze}. The same authors successively applied their approach to anomaly detection \cite{angelini2019privacy,angelini2019joint} and expanded MPOSE with the novel ISLD and ISLD-Additional-Sequences datasets. Conversely, Yan et al. \cite{yan2018spatial} first applied OpenPose to extract 2D poses from the Kinetics-400 RGB dataset \cite{kay2017kinetics} and used graph convolutions to capture spatial and temporal information.
Similarly, Shi et al. \cite{shi2019skeleton,shi2019two} applied graph convolutions to pose information using two streams that extract information from both joints and bones to represent the skeleton. Hao et al.\cite{HAO201913} introduced knowledge distillation and dense-connectivity to explore the spatiotemporal interactions between appearance and motion streams along different hierarchies. At the same time, Si et al. \cite{SI2020107511} put together a hierarchical spatial reasoning network and temporal stack learning network to extract discriminative spatial and temporal features.
On the other hand, Liu et al. \cite{liu2020disentangling} proposed an ensemble of two independent bone-based and joint-based models using a unified spatial-temporal graph convolutional operator. Conversely, Cho et al. \cite{cho2020self} first applied self-attention \cite{vaswani2017attention} to the skeletal-based HAR problem. More recently, Plizzari et al. \cite{plizzari2021skeleton}, inspired by Bello et al. \cite{bello2019attention}, employed self-attention to overcome the locality of the convolutions, again adopting a two-stream ensemble method, where self-attention is applied on both temporal and spatial information.

Unlike previous methodologies, this paper presents an architecture for HAR entirely based on the Transformer encoder, without any convolutional or recurrent layer. Moreover, we focus on verifying the suitability of our method for low-latency and real-time applications. For this reason, we introduce a new 2D pose-based dataset specifically designed for short-time HAR. Traditional datasets used by previous works, such as 3D NTU RGB+D \cite{shahroudy2016ntu,liu2019ntu}, or Kinetics-Skeleton \cite{kay2017kinetics,yan2018spatial}, include long temporal sequences which must be entirely scanned to make the correct classification. In contrast, the proposed MPOSE2021 dataset includes samples with a temporal duration of at most 30 frames. That makes it a new and more suitable benchmark to test the short-time and low-latency performance of HAR models.

\section{THE MPOSE2021 DATASET}
\label{mpose2021}
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{report_aver_conf.pdf}
    \caption{Comparison between OpenPose and PoseNet average confidence (aver\_conf) in the different MPOSE2021 sub-datasets. The detectors achieve different average confidence based on the considered precursor dataset.}
    \label{fig:mpose2021-aver-conf}
\end{figure}
In this section, MPOSE2021 is presented as an RGB-based dataset designed for short-time, pose-based HAR. As in \cite{angelini2018actionxpose, angelini2019privacy, angelini2019joint}, video data have been previously collected from popular HAR datasets (precursors), i.e. Weizmann \cite{gorelick2007actions}, i3DPost \cite{gkalelis2009i3dpost}, IXMAS \cite{weinland2006free}, KTH \cite{schuldt2004recognizing}, UTKinetic-Action3D (RGB only) \cite{xia2012view}, UTD-MHAD (RGB only) \cite{chen2015utd}, ISLD, and ISLD-Additional-Sequences \cite{angelini2018actionxpose}.

Due to the heterogeneity of actions across different datasets, labels are remapped to a list of 20 common classes. Actions that cannot be remapped accordingly are discarded. Therefore, precursor videos are divided into non-overlapping samples (clips) of 30 frames each whenever possible and retaining tail samples with more than 20 frames. A visual representation of the number of frames per sample for each sub-dataset of MPOSE2021 is shown in Fig. \ref{fig:mpose2021-frame-num}. The peculiarity of a reduced number of time steps contrasts with other publicly available datasets and stimulates the development of methodologies that require low latency to perform a prediction. That would largely benefit many real-world applications requiring real-time perception of the actions performed by humans nearby.

Subsequently, clips not containing a single action are discarded. Moreover, ambiguous clips are relabelled whenever possible or discarded otherwise. This process leads to 15429 samples, where each sample represents a single actor performing a single action. The total number of distinct actors in MPOSE2021 is 100, and the number of samples for each action is reported in Fig.\ref{fig:mpose2021-summary}, which also shows the distribution of precursor datasets.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{report_frame_num.pdf}
    \caption{MPOSE2021 sequence length distribution for the different sub-datasets. It is possible to notice how most of them present a mode of 30 frames per scene.}
    \label{fig:mpose2021-frame-num}
\end{figure}

OpenPose \cite{cao2019openpose} and PoseNet \cite{papandreou2018personlab} are used to extract landmarks from MPOSE2021 samples. The average confidence is computed for each sample as the mean across landmarks and frames. It turns out that the two detectors achieve different average confidences based on the considered precursor dataset. The box plot of Fig.\ref{fig:mpose2021-aver-conf} describes the comparison statistics.

Due to the significant sample heterogeneity and the high number of actors, three different training/testing splits are defined for MPOSE2021, i.e. Split1, Split2, and Split3, by randomly selecting 21 actors for testing and using the rest of them for training. This division makes the proposed dataset a challenging benchmark to effectively assess and compare the accuracy and robustness of different methodologies. Moreover, the suggested evaluation procedure requires testing a target model on each split using ten validation folds and averaging the obtained results. That makes it possible to produce statistics and reduces the possibility of overfitting the split testing set with an accurate choice of hyperparameters.

With MPOSE2021, we aim at providing an end-to-end and easy-to-use benchmark to robustly compare state-of-the-art methodologies for the short-time human action recognition task. We thus release a code repository\footnote{https://github.com/PIC4SeRCentre/MPOSE2021} to access the different levels of the dataset (video, RGB frames, 2D poses). Moreover, we open source a practical Python package to access, visualize and preprocess the poses with standard functions. The Python package can be easily installed with the command \hspace{0.02cm} \texttt{pip install mpose}.

\section{ACTION TRANSFORMER}
\label{actiontransformer}
In this section, we describe the architecture of the AcT network (Fig.\ref{fig:act_arch}) briefly recalling some preliminary concepts associated with the Transformer model\cite{vaswani2017attention}.

\subsection{AcT Architecture}
A video input sequence with $T$ frames of dimension $H \times W$ and $C$ channels $\textbf{X}_{rgb} \in \mathbb{R}^{T \times H \times W \times C}$ is pre-processed by a multi-person 2D pose estimation network
\begin{equation}
    \textbf{X}_{2Dpose} = F_{2Dpose}(\textbf{X}_{rgb})
\end{equation}
that extracts 2D poses of dimension $N \times T \times P$, where $N$ is the number of human subjects present in the frame and $P$ is the number of keypoints predicted by the network.
The Transformer architecture receives as input a 1D sequence of token embeddings, so each of the $N$ sequences of pose matrices $\textbf{\textit{X}}_{2Dpose} \in  \mathbb{R}^{T \times P}$ is separately processed by the AcT network. Nevertheless, at inference time, all detected poses in the video frame can be batch-processed by the AcT model, simultaneously producing a prediction for all $N$ subjects.
\begin{figure}[t]
    \includegraphics[width=\linewidth]{transformer_inside.png}
    \caption{Transformer encoder layer architecture (left) and schematic overview of a multi-head self-attention block (right). Input tokens go through L encoder layers and H self-attention heads.}
    \label{fig:act_inside}
\end{figure}
Firstly, the $T$ poses are mapped to a higher dimension $D_{model}$ using a linear projection map $\textbf{\textit{W}}^{l_{0}} \in \mathbb{R}^{P \times D_{model}}$. As in BERT \cite{devlin2018bert} and Vision Transformers \cite{dosovitskiy2020image, touvron2020training, berg2021keyword, caron2021emerging}, a trainable vector of dimension $D_{model}$ is added to the input $T$ sequence. This class token [CLS] forces the self-attention to aggregate information into a compact high-dimensional representation that separates the different action classes. Moreover, positional information is provided to the sequence with a learnable positional embedding matrix $\textbf{\textit{X}}_{pos} \in \mathbb{R}^{(T+1)\times D_{model}}$ added to all tokens.

The linearly projected tokens and [CLS] are fed to a standard Transformer encoder $F_{Enc}$ of L layers with a post-norm layer normalization \cite{vaswani2017attention, chen2018best}, obtaining

\begin{equation}
    \textbf{\textit{X}}^{L} = F_{Enc}(\textbf{\textit{X}}^{l_{0}}) = F_{Enc}([\textbf{\textit{x}}_{cls}^{l_0};\textbf{\textit{X}}_{2Dpose}] + \textbf{\textit{X}}_{pos})
\end{equation}
where $\textbf{\textit{X}}^{L}\in \mathbb{R}^{(T+1) \times D_{model}}$ is the overall representation produced by the Transformer encoder at its last layer. Finally, only the [CLS] token $\textbf{\textit{x}}_{cls}$ is fed into a linear classification head $\text{MLP}_{head}$ that performs the final class prediction

\begin{equation}
    \hat{\textbf{\textit{z}}} = \text{MLP}_{head}(\textbf{\textit{x}}_{cls}^{L})
\end{equation}

where $\hat{\textbf{\textit{z}}}$ is the output logit vector of the model. At training time, the supervision signal comes only from the [CLS] token, while all remaining $T$ tokens are the only input of the model. It is important to notice how the nature of the network makes it possible to accept a reduced number of frames as input even if trained with a fixed $T$. That gives an additional degree of freedom at inference time, making AcT more adaptive than other existing models.

The resulting network is a lightweight solution capable of predicting actions for multiple people in a video stream with high accuracy. The advantage of building on 2D pose estimations enables effective real-time performance with low latency and energy consumption.

\subsection{Transformer Architecture}
The Transformer encoder \cite{vaswani2017attention} is made of L layers with alternating H multi-head self-attention and feed-forward blocks. Dropout \cite{srivastava2014dropout}, Layernorm \cite{ba2016layer}, and residual connections are applied after every block. The overall sequence of blocks of a Transformer encoder is summarized on the left of Fig.\ref{fig:act_inside}.

Each feed-forward block is a multi-layer perceptron with two layers and GeLu \cite{hendrycks2016gaussian} non-linearity. The first layer expands the dimension from $D_{model}$ to $D_{mlp}=4\cdot D_{model}$ and applies the activation function. On the other hand, the second layer reduces the dimension back from $D_{mlp}$ to $D_{model}$.

Instead, the multi-head $\textbf{\textit{Q}}\textbf{\textit{K}}\textbf{\textit{V}}$ self-attention mechanism (MSA) is based on a trainable associative memory with key-value vector pairs.
For the l-th layer of the Transformer encoder and the h-th head, queries ($\textbf{\textit{Q}}$), keys ($\textbf{\textit{K}}$) and values ($\textbf{\textit{V}}$) are computed as $\textbf{\textit{Q}}=\textbf{\textit{X}}\textbf{\textit{W}}_Q$, $\textbf{\textit{K}}=\textbf{\textit{X}}\textbf{\textit{W}}_K$ and $\textbf{\textit{V}}=\textbf{\textit{X}}\textbf{\textit{W}}_V$ respectively, where $\textbf{\textit{W}}_Q$, $\textbf{\textit{W}}_K$ and $\textbf{\textit{W}}_V$ belong to $\mathbb{R}^{D_{model}\times D_{h}}$ , being $D_{h}$ the dimension of the attention head. So, for each self-attention head (SA) and element in the input sequence $\textbf{\textit{X}} \in \mathbb{R}^{T \times D_{model}}$ we compute a weighted sum over all values $\textbf{\textit{V}}$. The attention weights $A_{ij}$ are derived from the pairwise similarity between two elements of the sequence and their respective query $Q_i$ and key $K_j$ representations. Therefore, it is possible to compute the attention weights $\textbf{\textit{A}} \in \mathbb{R}^{T \times T}$ for the l-th layer as

\begin{table}[t]
\caption{Action Transformer parameters for the four version sizes. We fix $D_{model}/\text{H}=64$, linearly increasing H, $D_{mlp}$, and L in order to obtain different versions of the AcT network.}
\centering
\begin{tabular}{cccccc}
\toprule
\textbf{Model} & H & $D_{model}$ & $D_{mlp}$ & L & Parameters \\ \hline
AcT-$\mu$         & 1 & 64       & 256    & 4 & 227k          \\
AcT-S         & 2 & 128      & 256    & 5 & 1,040k        \\
AcT-M         & 3 & 192      & 256    & 6 & 2,740k        \\
AcT-L         & 4 & 256      & 512    & 6 & 4,902k        \\
\bottomrule
\end{tabular}
\label{tab:settings}
\end{table}

\begin{equation}
    \textbf{\textit{A}}=\text{Softmax}\left(\frac{\textbf{\textit{Q}}\textbf{\textit{K}}^{T}}{\sqrt{D_{h}}}\right)
\end{equation}
and the final weighted sum SA as
\begin{equation}
    \text{SA}(\textbf{\textit{X}})=\textbf{\textit{A}}\textbf{\textit{V}}
\end{equation}
We perform this SA operation for all the H heads of the l-th layer. We concatenate the results and linearly project the output tensor to the original dimension model as
\begin{equation}
    \text{MSA}(\textbf{\textit{X}})=[SA_{1}(\textbf{\textit{X}});SA_{2}(\textbf{\textit{X}});...;SA_{\text{H}}(\textbf{\textit{X}})]\textbf{\textit{W}}_{MSA}
\end{equation}

where $\textbf{\textit{W}}_{MSA} \in \mathbb{R}^{\text{H}\cdot D_{h}\times D_{model}}$.  All operations are schematized on the right side of Fig. \ref{fig:act_inside}. The input tensor $\textbf{\textit{X}}$ is firstly projected to $T\times \text{H} \cdot D_{h}$, and then a SA operation is performed to all H splits of the resulting projected tensor. Finally, all head outputs are concatenated and linearly projected back to $D_{model}$. So, the attention mechanism takes place in the time-domain, allowing a global class embedding representation by correlating different time windows.

In order to reduce the number of hyperparameters and linearly scale the dimension of AcT, we fix $D_{model}/\text{H}=64$, varying H, $D_{mlp}$, and L to obtain different versions of the network. A simple grid search using train and validation sets is performed to determine lower and upper bounds for the four parameters. In particular, in Table \ref{tab:settings}, we summarize the four AcT versions with their respective number of parameters. The four models (micro, small, medium, and large) differ for their increasing number of heads and layers, substantially impacting the number of trainable parameters.

\section{EXPERIMENTS}
\label{experiments}
This section describes the main experiments conducted to study the advantages of using a fully self-attentional model for 2D pose-based HAR. First, the four variants of AcT described in Section \ref{actiontransformer} are compared to existing state-of-the-art methodologies and baselines on MPOSE2021. Only for a specific comparison with ST-TR \cite{plizzari2021skeleton} and MS-G3D \cite{liu2020disentangling}, we use additional ensemble versions of our model named AcT-$\mu$ (x $n$), $n$ being the number of ensembled instances. Then, we further analyze the behavior of the network in order to get a visual insight of the attention mechanism and study the performance under a reduction of temporal information. Finally, we study model latency for all the designed architectures with two different CPU types, proving that AcT can easily be used for real-time applications.

\subsection{Experimental Settings}
In the following experiments, we employ both the OpenPose and PoseNet versions of the MPOSE2021 dataset. Either set of data has $T=30$ and $P=52$ or $68$ features, respectively. In particular, for OpenPose we follow the same preprocessing of \cite{angelini2018actionxpose} obtaining 13 keypoints with four parameters each (position $x,y$ and velocities $v_x,v_y$), while PoseNet samples contain 17 keypoints with the same information. The training set is composed of 9421 samples and 2867 for testing. The remaining 3141 instances are used as validation to find the most promising hyperparameters with a grid search analysis. All results, training, and testing code for the AcT model are open source and publicly available\footnote{https://github.com/PIC4SeRCentre/AcT}.

\begin{table}[t]
\scriptsize
\caption{Hyperparameters used in AcT experiments.}
\label{tab:hyperparams_act}
\centering
\begin{tabular}{ll}
\toprule
\multicolumn{2}{c}{\textbf{Training}} \\
\midrule
Training epochs & 350 \\
Batch size & 512 \\
Optimizer & AdamW \\
Warmup epochs & 40\% \\
Step epochs & 80\% \\
\bottomrule
\end{tabular}
\qquad
\begin{tabular}{ll}
\toprule
\multicolumn{2}{c}{\textbf{Regularization}} \\
\midrule
Weight decay & 1e-4 \\
Label smoothing & 0.1 \\
Dropout & 0.3 \\
Random flip & 50\% \\
Random noise $\sigma$ & 0.03 \\
\bottomrule
\end{tabular}
\end{table}

\begin{table*}[ht]
\centering
\caption{Benchmark of different models for short-time HAR on MPOSE2021 splits using OpenPose 2D skeletal representations.}
\begin{tabular}{@{}cccccccc@{}}
\toprule
\multicolumn{2}{c}{\textbf{MPOSE2021 Split}} & \multicolumn{2}{c}{\textbf{OpenPose 1}} & \multicolumn{2}{c}{\textbf{OpenPose 2}} & \multicolumn{2}{c}{\textbf{OpenPose 3}} \\ \midrule
\textbf{Model} & \textbf{Parameters} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} \\ \midrule
\textbf{MLP} & 1,334k & 82.66 ± 0.33 & 74.56 ± 0.56 & 84.41 ± 0.60 & 74.58 ± 1.00 & 83.48 ± 0.58 & 76.60 ± 0.77 \\
\textbf{Conv1D} & 4,037k & 88.18 ± 0.64 & 81.97 ± 1.40 & 88.93 ± 0.43 & 80.49 ± 0.95 & 88.67 ± 0.38 & 83.93 ± 0.58 \\
\textbf{REMNet \cite{angarano2021robust}} & 4,211k & 89.18 ± 0.51 & 84.20 ± 0.84 & 88.77 ± 0.35 & 80.29 ± 0.88 & 89.80 ± 0.59 & 86.18 ± 0.40 \\ \midrule
\textbf{ActionXPose \cite{angelini2018actionxpose}} & 509k & 87.60 ± 0.98 & 82.13 ± 1.50 & 88.42 ± 0.70 & 81.28 ± 1.40 & 89.96 ± 1.00 & 86.65 ± 1.60 \\
\textbf{MLSTM-FCN \cite{karim2018multi}} & 368k & 88.62 ± 0.74 & 83.55 ± 0.88 & 90.19 ± 0.68 & 83.84 ± 1.20 & 89.80 ± 0.94 & 87.33 ± 0.67 \\
\textbf{T-TR \cite{plizzari2021skeleton}} & 3,036k & 87.72 ± 0.87 & 81.99 ± 1.64 & 88.14 ± 0.53  & 80.23 ± 1.19  & 88.69 ± 0.95 & 85.03 ± 1.60 \\
\textbf{MS-G3D (J) \cite{liu2020disentangling}} & 2,868k & 89.90 ± 0.50 & 85.29 ± 0.98 & 90.16 ± 0.64 & 83.08 ± 1.10 & 90.39 ± 0.44 & 87.48 ± 1.20 \\

\midrule
\textbf{AcT-$\mu$} & 227k & 90.86 ± 0.36 & 86.86 ± 0.50 & 91.00 ± 0.24 & 85.01 ± 0.51 & 89.98 ± 0.47 & 87.63 ± 0.54 \\
\textbf{AcT-S} & 1,040k & 91.21 ± 0.48 & 87.48 ± 0.76 & 91.23 ± 0.19 & 85.66 ± 0.58 & 90.90 ± 0.87 & 88.61 ± 0.73 \\
\textbf{AcT-M} & 2,740k & \textbf{91.38 ± 0.32} & \textbf{87.70 ± 0.47} & 91.08 ± 0.48 & 85.18 ± 0.80 & 91.01 ± 0.57 & 88.63 ± 0.51 \\
\textbf{AcT-L} & 4,902k & 91.11 ± 0.32 & 87.27 ± 0.46 & \textbf{91.46 ± 0.42} & \textbf{85.92 ± 0.63} & \textbf{91.05 ± 0.80} & \textbf{89.00 ± 0.74} \\
\midrule
\textbf{ST-TR \cite{plizzari2021skeleton}} & 6,072k	& 89.20 ± 0.71 & 83.95 ± 1.11 & 89.29 ± 0.81 & 81.53 ± 1.39 & 90.49 ± 0.53 & 87.06 ± 0.70\\
\textbf{MS-G3D (J+B) \cite{liu2020disentangling}} & 5,735k & 91.13 ± 0.33 & 87.25 ± 0.50 & 91.28 ± 0.29 & 85.10 ± 0.50 & 91.42 ± 0.54 & 89.66 ± 0.55 \\

\textbf{AcT-$\mu$ (x2)} & 454k & 91.76 ± 0.29 & 88.27 ± 0.37 & 91.34 ± 0.40 & 86.88 ± 0.48 & 91.70 ± 0.57 & 88.87 ± 0.37 \\
\textbf{AcT-$\mu$ (x5)} & 1,135k & 92.43 ± 0.24 & 89.33 ± 0.31 & 91.55 ± 0.37 & 87.80 ± 0.39 & 92.63 ± 0.55 & 89.77 ± 0.35 \\
\textbf{AcT-$\mu$ (x10)} & 2,271k & \textbf{92.54 ± 0.21} & \textbf{89.79 ± 0.34} & \textbf{92.03 ± 0.33} & \textbf{88.02 ± 0.31} & \textbf{93.10 ± 0.53} & \textbf{90.22 ± 0.31} \\
\bottomrule
\end{tabular}
\label{tab:benchmark_openpose}
\end{table*}

\begin{figure*}[ht]
    \centering
    \includegraphics[width=\textwidth]{mean_balanced.png}
    \caption{Visual representation of the benchmark of different models for short-time HAR on MPOSE2021 splits using OpenPose 2D skeletal representations. For brevity and clearness, the average balanced accuracy on the three splits of MPOSE2021 is reported. The lines connect models that use the same methodology.}
    \label{fig:mean_balanced}
\end{figure*}

In Table \ref{tab:settings} are summarized all the hyperparameters for the four versions of the AcT architecture and in Table \ref{tab:hyperparams_act} all settings related to the training procedure.
The AdamW optimization algorithm \cite{loshchilov2017decoupled} is employed for all training with the same scheduling proposed in \cite{vaswani2017attention}, but with a step drop of the learning rate $\lambda$ to 1e-4 at a fixed percentage (80\%) of the total number of epochs. We employ the TensorFlow 2\footnote{https://www.tensorflow.org} framework to train the proposed network on a PC with 32-GB RAM, an Intel i7-9700K CPU, and an Nvidia 2080 Super GP-GPU. Following the previously defined benchmark strategy, the total training procedure for the four versions takes approximately 32 hours over the three different splits. We exploit publicly available code for what concerns other state-of-the-art models and use the same hyperparameters and optimizer settings described by the authors in almost all the cases. The only exception is made for learning rate, epochs number, and batch size to adapt the methodologies to our dataset and obtain better learning curves.

\begin{table*}[ht]
\centering
\caption{Benchmark of different models for short-time HAR on MPOSE2021 splits using PoseNet 2D skeletal representations.}
\begin{tabular}{@{}cccccccc@{}}
\toprule
\multicolumn{2}{c}{\textbf{MPOSE2021 Split}} & \multicolumn{2}{c}{\textbf{PoseNet 1}} & \multicolumn{2}{c}{\textbf{PoseNet 2}} & \multicolumn{2}{c}{\textbf{PoseNet 3}} \\ \midrule
\textbf{Model} & \textbf{Parameters} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} & \textbf{Accuracy {[}\%{]}} & \textbf{Balanced {[}\%{]}} \\ \midrule
\textbf{Conv1D} & 4,062k & 85.83 ± 0.71 & 79.96 ± 1.10 & 87.47 ± 0.35 & 78.51 ± 0.78 & 87.46 ± 0.67 & 81.31 ± 0.58 \\
\textbf{REMNet \cite{angarano2021robust}} & 4,269k & 84.75 ± 0.65 & 77.23 ± 0.94 & 86.17 ± 0.68 & 75.79 ± 1.30 & 86.31 ± 0.60 & 79.20 ± 0.79 \\ \midrule
\textbf{ActionXPose \cite{angelini2018actionxpose}} & 509k & 75.98 ± 0.72 & 64.47 ± 1.10 & 79.94 ± 1.10 & 67.05 ± 1.40 & 77.34 ± 1.40 & 66.86 ± 1.40\\
\textbf{MLSTM-FCN \cite{karim2018multi}} & 368k & 76.17 ± 0.84 & 64.75 ± 1.10 & 79.04 ± 0.72 & 65.62 ± 1.40 & 77.84 ± 1.30 & 67.05 ± 1.20 \\

\midrule
\textbf{AcT-$\mu$} & 228k & 86.66 ± 1.10 & 81.56 ± 1.60 & 87.21 ± 0.99 & 79.21 ± 1.60 & 87.75 ± 0.53 & 82.99 ± 0.87 \\
\textbf{AcT-S} & 1,042k & \textbf{87.63 ± 0.52} & \textbf{82.54 ± 0.87} & 88.48 ± 0.57 & 81.53 ± 0.68 & 88.49 ± 0.65 & 83.63 ± 0.99 \\
\textbf{AcT-M} & 2,743k & 87.23 ± 0.48 & 82.10 ± 0.66 & \textbf{88.50 ± 0.51} & \textbf{81.79 ± 0.44} & \textbf{88.70 ± 0.57} & \textbf{83.92 ± 0.96} \\
\bottomrule
\end{tabular}
\label{tab:benchmark_posenet}
\end{table*}

\subsection{Action Recognition on MPOSE2021}
We extensively experiment on MPOSE2021 considering some baselines, common HAR architectures, and our proposed AcT models. We report the mean and standard deviation of 10 models trained using different validation splits to obtain statistically relevant results. The validation splits are constant for all the models and correspond to 10\% of the train set, maintaining the same class distribution. The benchmark is executed for both OpenPose and PoseNet data and repeated for all the three train/test splits provided by MPOSE2021. The baselines chosen for the benchmark are a Multilayer Perceptron (MLP), a fully convolutional model (Conv1D), and REMNet, which is a more sophisticated convolutional network with attention and residual blocks proposed in \cite{angarano2021robust} for time series feature extraction. In particular, the MLP is designed as a stack of three fully connected (FC) layers with 512 neurons each, followed by a dropout layer and a final FC layer with as many output nodes as classes. Instead, the Conv1D model is built concatenating five 1D convolutional layers with 512 filters alternated with batch normalization stages and followed by a global average pooling operator, a dropout layer, and an FC output stage as in the MLP. Finally, the configuration used for REMNet consists of two Residual Reduction Modules (RRM) with a filter number of 512, followed by dropout and the same FC output layer as in the other baselines.

Regarding state-of-the-art comparisons, four popular models used for multivariate time series classification, and in particular HAR, are reproduced and tested. Among those, MLSTM-FCN \cite{karim2018multi} combines convolutions, spacial attention, and an LSTM block, and its improved version ActionXPose \cite{angelini2018actionxpose} uses additional preprocessing, leading the model to exploit more correlations in data and, hence, be more robust against noisy or missing pose detections. On the other hand, MS-G3D \cite{liu2020disentangling} uses spatial-temporal graph convolutions to make the model aware of spatial relations between skeleton keypoints, while ST-TR \cite{plizzari2021skeleton} joins graph convolutions with Transformer-based self-attention applied to both space and time. As the last two solutions also propose a model ensemble, these results are further compared to AcT ensembles made of 2, 5, and 10 single-shot models. We also report the achieved balanced accuracy for each model and use it as the primary evaluation metric to account for the uneven distribution of classes.

The results of the experimentation for OpenPose are reported in Table \ref{tab:benchmark_openpose} and, in synthesis, in Fig.\ref{fig:mean_balanced}. The fully convolutional baseline strongly outperforms the MLP, while REMNet proves that introducing attention and residual blocks further increases accuracy. As regards MLSTM-FCN and ActionXPose, it is evident that explicitly modeling both spatial and temporal correlations, made possible by the two separate branches, slightly improves the understanding of actions with respect to models like REMNet. MS-G3D, in its joint-only (J) version, brings further accuracy improvement by exploiting graph convolutions and giving the network information on the spatial relationship between keypoints. On the other hand, ST-TR shows performance comparable to all other single-shot models, despite, as MS-G3D, taking advantage of graph information.

The proposed AcT model demonstrates the potential of pure Transformer-based architectures, as all four versions outperform other methodologies while also showing smaller standard deviations. Moreover, even the smallest AcT-$\mu$ (227k parameters) is able to extract general and robust features from temporal correlations in sequences. Increasing the number of parameters, a constant improvement in balanced accuracy can be observed for splits 3, while split 1 and 2 present oscillations. The difference between splits reflects how much information is conveyed by training sets and how much the model can learn from that. So, it is evident that AcT scales best on split three because it presents complex correlations that a bigger model learns more easily. On the contrary, it seems AcT-$\mu$ is able to extract almost all the information relevant for generalization from split 2, as the accuracy only slightly increases going towards more complex models.

\begin{figure*}[t]
    \centering
    \includegraphics[width=\textwidth]{attention_mask_big.png}
    \caption{Self-attention weights $\textbf{\textit{A}}$ of MPOSE2021 test samples. $(l,t,p)$ represents the AcT-M l-th layer, the true label and the prediction respectively. The three rightmost columns show three attention maps of a failed prediction and the other columns are from correct classifications. It is clear from all examples how the model focuses on certain particular frames of the series in order to extract a global representation of the scene.}
    \label{fig:attention_vis}
\end{figure*}

As explained in Section \ref{relatedworks}, ST-TR exploits an ensemble of two networks modeling spatial and temporal sequence correlations, respectively. Moreover, MS-G3D leverages further information such as skeleton graph connections and the position of bones in one of the ensembled networks. Ensembles are very effective in reducing model variance and enhancing performance by exploiting independent representations learned by each network, so it is unfair to compare them with single architectures. For this reason, we create three ensemble versions of AcT-$\mu$ to have an even confront, with 2, 5, and 10 instances, respectively. To compute ensemble predictions, we average the output logits of the network instances before passing through a softmax function. The results reported at the bottom of Table \ref{tab:benchmark_openpose} show that AcT-$\mu$ (x2) outperforms MS-G3D (J+B) in all the benchmarks except for balanced accuracy in split 3, despite having less than one-tenth of its parameters. Finally, the ensembles AcT-$\mu$ (x5) and AcT-$\mu$ (x10), made of 5 and 10 instances respectively, achieve even higher accuracy on all the splits with only around 1 to 2 million parameters. That proves how the balancing effect of the ensemble enhances model predictions even without feeding the network additional information.

As PoseNet data is mainly dedicated to real-time and Edge AI applications, only the models designed for this purpose have been considered in the benchmark, excluding MS-G3D, ST-TR, and AcT-L. In general, the results give similar insights. The tested models are the same as the previous case, with all the necessary modifications given by the different input formats. In the MLP case, however, performance seriously degrades as networks strongly tend to overfit input data after a small number of epochs, so the results are not included in Table \ref{tab:benchmark_posenet}. That is caused by the fact that PoseNet is a lighter methodology developed for Edge AI, and hence noisy and even missing keypoint detections are more frequent. That results in less informative data and emphasizes the difference between sequences belonging to different sub-datasets, confusing the model and inducing it to learn very specific but unusable features. The MLP is too simple and particularly prone to this kind of problem. Naturally, all the models are affected by the same problem, and the balanced accuracy on PoseNet is generally lower. The same considerations made for OpenPose apply in this case, where AcT outperforms all the other architectures and demonstrates its ability to give an accurate and robust representation of temporal correlations. Also, it is interesting to notice that Conv1D performs better than REMNet, proving to be less prone to overfitting, and that standard deviations are more significant than in the OpenPose case.

\subsection{Model Introspection}
In order to have an insight into the frames of the sequence the AcT model attends to, we extract the self-attention weights at different stages of the network. In Fig.\ref{fig:attention_vis}, MPOSE2021 test samples are propagated through the AcT-M model, and attention weights $\textbf{\textit{A}}$ of the three distinct heads are presented. It can be seen that the model pays attention to specific frames of the sequence when a specific gesture defines the action. On the other hand, attention is much more spread across the different frames for more distributed actions such as walking and running. Moreover, it is clear how the three heads mostly focus on diverse frames of the sequence. Finally, the rightmost columns show three attention maps of failed predictions. In these last cases, attention weights are less coherent, and the model cannot extract a valid global representation of the scene.

\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{attention_frames.png}
    \caption{Self-attention of the [CLS] token, computed as the normalized sum of the last layer of the different heads. Scores give a direct insight into the frames exploited by AcT to produce the classification output. The example clearly shows how bending positions are more insightful for the network to predict the jumping-in-place action. In the image, the attention score defines the skeleton alpha channel.}
    \label{fig:attention_frames}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{drop_frames.png}
    \caption{AcT-M balanced accuracy with an incremental reduction of temporal information. Due to the intrinsic nature of the network, it is possible to reduce the number of temporal steps without a retraining or any kind of explicit adaptation.}
    \label{fig:drop_frames}
\end{figure}

Instead, in Fig.\ref{fig:attention_frames}, the last-layer self-attention scores of the [CLS] token are shown together with the RGB and skeleton representations of the scene. The scores are computed as the normalized sum of the three attention heads and give a direct insight into the frames exploited by AcT to produce the classification output. It can be seen that bent poses are much more informative for the model to predict the jumping-in-place action.

Moreover, we analyze the behavior of the network under a progressive reduction of temporal information. That can be easily done without retraining due to the intrinsic nature of AcT. In Fig.\ref{fig:drop_frames}, we present how the test-set balanced accuracy is affected by frame dropping. The two curves show a reduction starting from the beginning and the end of the temporal sequence, respectively. It is interesting to notice how the performance of AcT degrades with an almost linear trend. That highlights the robustness of the proposed methodology and demonstrates the possibility of adapting the model to applications with different temporal constraints.

Finally, we also study the positional embeddings of the AcT-M model by analyzing their cosine similarity, as shown in Fig.\ref{fig:cosine}. Very nearby position embeddings demonstrate a high level of similarity, and distant ones are orthogonal or in the opposite direction. This pattern is constant for all $T$ frames of the sequence, highlighting how actions are not particularly localized and that relative positions are essential for all the frames.

\subsection{Latency Measurements}
We test the performance of all the considered models for real-time applications. To do so, we use the TFLite Benchmark\footnote{https://www.tensorflow.org/lite/performance/measurement} tool, which allows running TensorFlow Lite models on different computing systems and collecting statistical results on latency and memory usage. In our case, two CPUs are employed to measure model speed both on a PC and a mobile phone: an Intel i7-9700K for the former and the ARM-based HiSilicon Kirin 970 for the latter. In both experiments, the benchmark executes 10 warm-up runs followed by 100 consecutive forward passes, using 8 threads.

\begin{figure}[ht]
\centering
    \includegraphics[scale=0.65]{cosine_similarity.pdf}
    \vspace{-20pt}
    \caption{Cosine similarities of the learned $T$ position embeddings of AcT-M model.}
    \label{fig:cosine}
\end{figure}

\begin{figure}[ht]
    \centering
    \includegraphics[width=\columnwidth]{latency.png}
    \caption{Study of the latency of different tested models on a high-performance Intel CPU and on a mobile phone equipped with an ARM-based CPU.}
    \label{fig:latency}
\end{figure}

The results of both tests are reported in Fig. \ref{fig:latency}, where only the MLP has been ignored because, despite being the fastest-running model, its accuracy results are much lower than its competitors. The graph shows the great computational efficiency of Transformer-based architectures, whereas convolutional and recurrent networks result in heavier CPU usage. Indeed, in the Intel i7 case, REMNet achieves almost the same speed as AcT-S, but its accuracy is 2\% lower. Moreover, AcT-$\mu$ is able to outperform REMNet, running at over four times its speed. MLSTM-FCN and ActionXPose, being smaller models, achieve lower latencies than the baselines: the former stays between AcT-$\mu$ and Act-S, while the latter performs similarly to AcT-S. Those results are remarkable but still outperformed by AcT-$\mu$ both on accuracy and speed.

The difference with the baselines is even more evident on the ARM-based chip, as convolutional architectures seem to perform poorly on this kind of hardware. Indeed, REMNet and Conv1D run as fast as AcT-M with significantly lower accuracies, and AcT-$\mu$ is ten times quicker. Nothing changes for what concerns MLSTM-FCN and ActionXPose, less accurate and three times slower than AcT-$\mu$.

\section{CONCLUSION}
\label{conclusion}
In this paper, we explored the direct application of a purely Transformer-based network to human action recognition. We introduced the AcT network, which significantly outperforms commonly adopted models for HAR with a simple and fully self-attentional architecture. In order to limit computational and power requests, building on previous HAR and pose estimation research, the proposed methodology exploits 2D skeletal representations of short time sequences, providing an accurate and low latency solution for real-time applications. Moreover, we introduced MPOSE2021, a large-scale open-source dataset for short-time human action recognition, as an attempt to build a formal benchmark for future research on the topic. Extensive experimentation with our methodology clearly demonstrates the effectiveness of AcT and poses the basis for meaningful impact on many practical computer vision applications. In particular, the remarkable efficiency of AcT could be exploited for Edge AI, achieving good performance even on computationally limited devices. Future work may further investigate the AcT architecture with 3D skeletons and long sequence inputs. Moreover, relational information between points of the estimated pose are currently scarcely exploited by our methodology. So, future investigations may attempt to include skeleton graphs as prior knowledge in the positional embedding of the network without impacting latency.

{\small
\bibliographystyle{ieee}

\section{INTRODUCTION}

Your goal is to simulate, as closely as possible, the usual appearance of typeset
 papers. This document provides an example of the desired layout and contains
 information regarding desktop publishing format, type sizes, and type faces.

\subsection{Full-Size Camera-Ready (CR) Copy}

If you have desktop publishing facilities, (the use of a computer to aid
 in the assembly of words and illustrations on pages) prepare your CR paper
  in full-size format, on paper 21.6 x 27.9 cm (8.5 x 11 in or 51 x 66 picas).
  It must be output on a printer (e.g., laser printer) having 300 dots/in, or
  better, resolution. Lesser quality printers, such as dot matrix printers,
   are not acceptable, as the manuscript will not reproduce the desired quality.

\subsubsection{Typefaces and Sizes:} There are many different typefaces and a large
variety of fonts (a complete set of characters in the same typeface, style,
 and size). Please use a proportional serif typeface such as Times Roman,
 or Dutch. If these are not available to you, use the closest typeface you
  can. The minimum typesize for the body of the text is 10 point. The minimum
  size for applications like table captions, footnotes, and text subscripts
  is 8 point. As an aid in gauging type size, 1 point is about 0.35 mm (1/72in).
   Examples are as follows:

\subsubsection{Format:} In formatting your original 8.5" x 11" page, set top and
bottom margins to 25 mm (1 in or 6 picas), and left and right margins
to about 18 mm (0.7 in or 4 picas). The column width is 88 mm (3.5 in or 21 picas).
 The space between the two columns is 5 mm(0.2 in or 1 pica). Paragraph
 indentation is about 3.5 mm (0.14 in or 1 pica). Left- and right-justify your
 columns. Cut A4 papers to 28 cm. Use either one or two spaces between sections,
 and between text and tables or figures, to adjust the column length.
  On the last page of your paper, try to adjust the lengths of the
  two-columns so that they are the same. Use automatic hyphenation and
   check spelling. Either digitize or paste your figures.

\begin{table}
\caption{An Example of a Table}
\label{table_example}
\begin{center}
\begin{tabular}{|c||c|}
\hline
One & Two\\
\hline
Three & Four\\
\hline
\end{tabular}
\end{center}
\end{table}

\section{UNITS}

Metric units are preferred for use in IEEE publications in light of their
international readership and the inherent convenience of these units in many fields.
In particular, the use of the International System of Units (SI Units) is advocated.
 This system includes a subsystem the MKSA units, which are based on the
 meter, kilogram, second, and ampere. British units may be used as secondary units
 (in parenthesis). An exception is when British units are used as identifiers in trade,
 such as, 3.5 inch disk drive.

\addtolength{\textheight}{-3cm}

\section{ADDITIONAL REQUIREMENTS}

\subsection{Figures and Tables}

Position figures and tables at the tops and bottoms of columns.
Avoid placing them in the middle of columns. Large figures and tables
may span across both columns. Figure captions should be below the figures;
 table captions should be above the tables. Avoid placing figures and tables
  before their first mention in the text. Use the abbreviation ``Fig. 1'',
  even at the beginning of a sentence.
Figure axis labels are often a source of confusion.
Try to use words rather then symbols. As an example write the quantity ``Inductance",
 or ``Inductance L'', not just.
 Put units in parentheses. Do not label axes only with units.
 In the example, write ``Inductance (mH)'', or ``Inductance L (mH)'', not just ``mH''.
 Do not label axes with the ratio of quantities and units.
 For example, write ``Temperature (K)'', not ``Temperature/K''.

\subsection{Numbering}

Number reference citations consecutively in square brackets \cite{c1}.
 The sentence punctuation follows the brackets \cite{c2}.
 Refer simply to the reference number, as in \cite{c3}.
 Do not use ``ref. \cite{c3}'' or ``reference \cite{c3}''.
Number footnotes separately in superscripts\footnote{This is a footnote}
Place the actual footnote at the bottom of the column in which it is cited.
Do not put footnotes in the reference list.
Use letters for table footnotes (see Table I).

\subsection{Abbreviations and Acronyms}

Define abbreviations and acronyms the first time they are used in the text,
even after they have been defined in the abstract. Abbreviations such as
IEEE, SI, CGS, ac, dc, and rms do not have to be defined. Do not use
abbreviations in the title unless they are unavoidable.

\subsection{Equations}

Number equations consecutively with equation numbers in parentheses flush
 with the right margin, as in (1). To make your equations more compact
 you may use the solidus (/), the exp. function, or appropriate exponents.
  Italicize Roman symbols for quantities and variables, but not Greek symbols.
   Use a long dash rather then hyphen for a minus sign. Use parentheses to avoid
    ambiguities in the denominator.
Punctuate equations with commas or periods when they are part of a sentence:
$$\Gamma_2 a^2 + \Gamma_3 a^3 + \Gamma_4 a^4 + ... = \lambda \Lambda(x),$$
where $\lambda$ is an auxiliary parameter.

Be sure that the symbols in your equation have been defined before the
equation appears or immediately following.
Use ``(1),'' not ``Eq. (1)'' or ``Equation (1),''
except at the beginning of a sentence: ``Equation (1) is ...''.

\begin{figure}[thpb]
  \centering

  \caption{Inductance of oscillation winding on amorphous magnetic core versus DC bias magnetic field}
  \label{figurelabel}
\end{figure}

\section{CONCLUSIONS AND FUTURE WORKS}

\subsection{Conclusions}

This is a repeat.
Position figures and tables at the tops and bottoms of columns.
Avoid placing them in the middle of columns. Large figures and tables
may span across both columns. Figure captions should be below the figures;
 table captions should be above the tables. Avoid placing figures and tables
  before their first mention in the text. Use the abbreviation ``Fig. 1'',
  even at the beginning of a sentence.
Figure axis labels are often a source of confusion.
Try to use words rather then symbols. As an example write the quantity ``Inductance",
 or ``Inductance L'', not just.
 Put units in parentheses. Do not label axes only with units.
 In the example, write ``Inductance (mH)'', or ``Inductance L (mH)'', not just ``mH''.
 Do not label axes with the ratio of quantities and units.
 For example, write ``Temperature (K)'', not ``Temperature/K''.

\subsection{Future Works}

This is a repeat.
Position figures and tables at the tops and bottoms of columns.
Avoid placing them in the middle of columns. Large figures and tables
may span across both columns. Figure captions should be below the figures;
 table captions should be above the tables. Avoid placing figures and tables
  before their first mention in the text. Use the abbreviation ``Fig. 1'',
  even at the beginning of a sentence.
Figure axis labels are often a source of confusion.
Try to use words rather then symbols. As an example write the quantity ``Inductance",
 or ``Inductance L'', not just.
 Put units in parentheses. Do not label axes only with units.
 In the example, write ``Inductance (mH)'', or ``Inductance L (mH)'', not just ``mH''.
 Do not label axes with the ratio of quantities and units.
 For example, write ``Temperature (K)'', not ``Temperature/K''.

\subsection{Future Works}

This is a repeat.
Position figures and tables at the tops and bottoms of columns.
Avoid placing them in the middle of columns. Large figures and tables
may span across both columns. Figure captions should be below the figures;
 table captions should be above the tables. Avoid placing figures and tables
  before their first mention in the text. Use the abbreviation ``Fig. 1'',
  even at the beginning of a sentence.
Figure axis labels are often a source of confusion.
Try to use words rather then symbols. As an example write the quantity ``Inductance",
 or ``Inductance L'', not just.
 Put units in parentheses. Do not label axes only with units.
 In the example, write ``Inductance (mH)'', or ``Inductance L (mH)'', not just ``mH''.
 Do not label axes with the ratio of quantities and units.
 For example, write ``Temperature (K)'', not ``Temperature/K''.

\section{ACKNOWLEDGMENTS}

The authors gratefully acknowledge the contribution of reviewers' comments, etc. (if desired). Put sponsor acknowledgments in the unnumbered footnote on the first page.

References are important to the reader; therefore, each citation must be complete and correct. If at all possible, references should be commonly available publications.

{\small
\bibliographystyle{ieee}