\section{Introduction}

\IEEEPARstart{I}{mage} captioning is one of the important tasks in the intersection of vision and language, which aims to automatically describe given images with natural language captions. Some recent studies have achieved superior performance and are even comparable to humans under certain circumstances \cite{anderson2018bottom,zhang2018high,yu2021dual}. However, these traditional image captioning models perform unsatisfactorily when describing scenes containing text. As shown in Case (a) (cf. \cref{fig:intro}), ``a bottle of Coca-Cola'' is obviously specific than ``a bottle''. To solve this defect, Sidorov \textit{et al.} \cite{sidorov2020textcaps} propose text-based image captioning and collect a high-quality dataset TextCaps, in which captions contain scene text from the~images.
\begin{figure}[t]
  \centering

   \includegraphics[width=1.0\linewidth]{./Intro5.pdf}
   \caption{Our DEVICE significantly improves the quality of captions. In Case a, with depth information and a depth-enhanced feature updating module, DEVICE correctly models 3D relationships between scene texts. With a salient visual object concepts extractor and a semantic-guided alignment module, DEVICE generates more accurate and comprehensive captions, cf. Case b. Scene text is represented in \textcolor[rgb]{0,0.69,0.23}{Green}, and objects are represented in \textcolor[rgb]{0,0.76,0.98}{Blue}.}
   \label{fig:intro}
\end{figure}

To utilize the scene text in pictures, extracting Optical Character Recognition (OCR) tokens from given images is the precondition. M4C-Captioner\cite{sidorov2020textcaps} utilizes Rosetta\cite{borisyuk2018rosetta} OCR system to obtain OCR tokens, and then employ a multimodal transformer directly to model the relationships between visual objects and OCR tokens. However, it is inappropriate to treat all OCR tokens equally, because only part of OCR tokens are crucial to the understanding of the scene. Wang \textit{et al.} \cite{wang2021confidence} creatively introduce a concept of OCR confidence to choose the noteworthy OCR tokens, which has greatly alleviated this issue. Then Wang \textit{et al.}\cite{wang2020multimodal, wang2021improving} are committed to improving the spatial relationship construct methods for OCR tokens, which have been proven effective, and the proposed model LSTM-R\cite{wang2021improving} achieves the state-of-the-art performance. However, current methods are not accurate and comprehensive enough when describing the scenes, which can be mainly manifested in the following two-fold.

Firstly, the real-world is three-dimensional (3D). However, current studies\cite{sidorov2020textcaps,wang2021confidence,wang2020multimodal,wang2021improving, yang2021tap} only exploit the spatial relationships of OCR tokens at the plane-level, which leads to visual location information being utilized incompletely, and even results in generating inaccurate scene text. As illustrated in the Case (a) of \cref{fig:intro}, CNMT\cite{wang2021confidence} mistakenly regards ``victorian'' and ``coca'' as closely related OCR tokens in the same plane, which are irrelevant and actually one behind the other. Naturally, we consider that introducing depth information based on two-dimensional (2D) information can simulate real-world geometric information for improving the accuracy of scene text in captions.

Secondly, scenes contain abundant but complex visual entities (visual objects and scene text), existing methods suffer from coarse-grained descriptions of visual objects and the absence of crucial visual entities in captions. As shown in the Case (b) on the right of \cref{fig:intro}, CNMT\cite{wang2021confidence} generates the coarse-grained word ``person'' for the visual object ``athlete". Besides, CNMT ignores an important entity ``bike'', causing the OCR token ``spinner'' on it cannot be utilized effectively. One possible reason for the absence of ``bike'' is that the visual features of mutilated objects are not conducive to modeling. Intuitively, we consider that the semantic information of salient visual object concepts can alleviate coarse-grained and partial captioning.

To tackle the aforementioned issues, we propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps. It mainly comprises a reading module, a depth-enhanced feature updating module, a salient visual object concepts extractor, a semantic-guided alignment module, and a multimodal transformer. Concretely, in the reading module, we employ a monocular depth estimator BTS\cite{lee2019big} to generate depth maps, then we extract depth values for visual objects and OCR tokens, respectively. The depth-enhanced feature updating module is designed to optimize the visual features of OCR tokens, which is beneficial for 3D geometry relationship construction. In the salient visual object concepts semantic extractor, we first employ a pre-trained visual-language model CLIP\cite{radford2021learning} as a filter to generate $K$ concepts most related to the given picture. Then we adopt the semantic information of visual object concepts as augmentations to improve the overall expressiveness of our model. To utilize visual concepts more efficiently, we introduce the Semantic-guided Alignment Module to interact aligned visual concepts with corresponding scene text. Subsequent experiments in \cref{sec4} demonstrate the effectiveness of our proposed DEVICE.

Our main contributions can be summarized as follows:
\begin{itemize}
\item We propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps. DEVICE performs better than previous methods as it improves the accuracy and completeness of captions.
\item We devise a Depth-enhanced Feature Updating Module (DeFUM) to enhance the 3D spatial relationship between OCR tokens, which effectively helps the model evaluate the relevance of OCR tokens. To our knowledge, we are the first to apply depth information to the text-based image captioning task.
\item We introduce semantic information of salient visual objects by the Visual Object Concepts Extractor (VOC), and we construct a Semantic-guided Alignment Module (SgAM) to improve the availability of visual concepts. Our DEVICE is capable of generating fine-grained objects in captions and better understanding holistic scenes.
\item Our DEVICE outperforms the state-of-the-art models on the TextCaps test set, inspiringly boosting CIDEr-D from 100.8 to 110.0.
\end{itemize}
\section{Related Work}
\subsection{Text-based Image Captioning}
Image captioning task has made a lot of progress in recent years\cite{vinyals2015show,anderson2018bottom,huang2019attention,huang2021image,cornia2020meshed}, but most of them perform poorly facing images with scene text. Text-based image captioning was naturally born to alleviate this problem, which was proposed by Sidorov \textit{et al.}\cite{sidorov2020textcaps}. The authors introduced the Textcaps dataset and a multimodal Transformer-based\cite{vaswani2017attention} baseline model M4C-Captioner. The M4C-Captioner was modified from a Text-VQA\cite{singh2019towards,biten2022latr,jin2022token,zeng2021beyond} model M4C\cite{hu2020iterative}, and can encode both image and OCR tokens to generate more property captions. However,  this paradigm suffers from a lack of diversity and information integrity. Zhang \textit{et al.}\cite{zhang2022magic} and Xu \textit{et al.}\cite{xu2021towards} were committed to solving this problem by introducing additional captions. Although introducing multiple captions alleviates the problem of missing scene information, it is inherently hard to summarize the complex entities of a scene within one caption well. Apart from this, M4C-Captioner is less than satisfactory performed on Textcaps due to insufficient use of visual and semantic information of images. To distinguish misidentified OCR recognition results and pick out the crucial OCR tokens, Wang \textit{et al.}\cite{wang2021confidence} introduced confidence embedding and alleviated the adverse effects of inaccurate OCR tokens to a certain extent. Yang \textit{et al.}\cite{yang2021tap} designed a pre-training model that can exploit OCR tokens and achieved strong results on multiple tasks, including text-based image captioning. Wang \textit{et al.}\cite{wang2020multimodal,wang2021improving} and Tang \textit{et al.}\cite{tang2022ocr} employ the 2D spatial relationships to measure correlations between OCR tokens. Nevertheless, we find that simply leveraging two-dimensional spatial relationships may cause incorrect captions. Therefore, we ameliorate this problem by utilizing depth information.

\subsection{Depth Information For Multimodal Tasks}
Monocular depth estimation is a challenging problem of computer vision, whose purpose is to use depth values to measure the distance of objects relative to the camera\cite{couprie2013indoor,geiger2012we,lee2019big}. With depth estimation technology, significant improvements have been achieved on many multimodal tasks\cite{banerjee2021weakly, wang2018look, liao2021scene}. In the visual question answer (VQA) task, Banerjee \textit{et al.}\cite{banerjee2021weakly} and Liu \textit{et al.}\cite{9830073} proved that exploiting three-dimensional relationships by depth information could enhance the spatial reasoning ability of VQA models. Apart from this, Qiu \textit{et al.}\cite{qiu20203d} and Liao \textit{et al.}\cite{liao2021scene} point out that ignoring the depth information could tremendously place restrictions on applications of scene change captioning in the real world. At the same time, we notice that text-based image captioning is facing the same problem. Therefore, we improve the 3D relationship modeling method to apply in text-based image captioning effectively.

\begin{figure*}[ht]
  \centering

   \includegraphics[width=0.96\linewidth]{./Model3.pdf}
   \caption{Overview of our DEpth and VIsual ConcEpts Aware Transformer for TextCaps (DEVICE). In the reading module, we first extract visual objects and OCR tokens along with their depth values. Then a depth-enhanced feature updating module is constructed to enhance OCR appearance features with the help of depth values. After that, DEVICE utilizes depth information and plane geometry relationships to construct 3D positional relationships. Semantic information of salient visual object concepts promotes fine-grained and holistic captioning, and the proposed semantic-guided alignment module improves the model's efficiency to utilize visual concepts. Words in \textcolor[rgb]{0,0.76,0.98}{Blue}, \textcolor[rgb]{0,0.75,0.23}{Green} and \textcolor[rgb]{0.95,0.65,0}{Orange} represent objects, scene text, and visual concepts, respectively.}
   \label{fig:model}
\end{figure*}

\section{Approach}

\subsection{Overview}

Given an image \textit{I}, text-based image captioning models aim to automatically generate caption $C=[y_0, y_1, ... , y_n]$ based on scene text \textit{S} in the image.

As depicted in \cref{fig:model}, our DEVICE mainly consists of a reading module, a depth-enhanced feature updating module, a salient visual object concepts extractor, a semantic-guided alignment module, a multimodal transformer, and a sentence decoder. Concretely, the input image is first put into an object detector and two OCR detectors to extract \textit{N} visual objects and \textit{M} OCR tokens along with their 2D regional coordinates. Then we utilize a pre-trained depth estimation model to generate the pixel-level depth map \textit{D} and construct 3D coordinates of visual entities. After that, we update the visual features of OCR tokens with the depth-enhanced feature updating module. Depth information is included in the coordinates of both scene text and objects, hence the multimodal transformer could establish 3D relations between visual entities. Meanwhile, to generate salient visual object concepts for each image, we employ a pre-trained cross-modal tool CLIP\cite{radford2021learning} to match the top-\textit{K} related visual concepts (e.g., ``Van'' and ``Vehicle'') in vocabulary. Then we propose the semantic-guided alignment module to integrate the information of aligned visual concepts into the representation of relevant OCR tokens. Ultimately, the multimodal transformer and sentence decoder generate captions in the auto-regressive paradigm.
\subsection{Multimodal Embedding}
First of all, we exploit pre-trained Faster R-CNN\cite{ren2015faster} model to get the bounding box coordinates $\{b^{obj}_n\}_{n=1:N}$ of objects $\{a^{obj}_n\}_{n=1:N}$. To ensure high quality OCR tokens, following LSTM-R\cite{wang2021improving}, we apply external OCR systems\cite{borisyuk2018rosetta,Googleocr} to obtain both OCR tokens $\{a^{ocr}_m\}_{m=1:M}$ and corresponding bounding box coordinates $\{b^{ocr}_m\}_{m=1:M}$, respectively. After that, we extract the $d$-dim appearance features $\{x^{of}_n\}_{n=1:N}$ of objects and appearance features $\{x^{tf}_m\}_{m=1:M}$ of OCR tokens by Faster R-CNN\cite{ren2015faster}. To generate depth maps of images in the TextCaps dataset, we utilize a monocular depth estimation model BTS\cite{lee2019big}.

\textbf{Depth values.}
To measure the relative distance of objects and OCR tokens to the observer, we employ a pre-trained depth estimation model BTS\cite{lee2019big} to generate depth maps. Subsequently, we unify the values of the depth maps to the range of 0 to 255. For object $a^{obj}_n$ and OCR token $a^{ocr}_m$, we count the grayscale distribution of bounding box region of $a^{obj}_n$ and $a^{ocr}_m$ in the depth map at pixel scale. We take the gray values with the highest frequency as the depth values for object $a^{obj}_n$ and OCR token $a^{ocr}_m$ (cf. \cref{fig:depth}). We denote $\{dv^{obj}_n\}_{n=1:N}$ as the depth values for $\{a^{obj}_n\}_{n=1:N}$ and $\{dv^{ocr}_m\}_{m=1:M}$ for $\{a^{ocr}_m\}_{m=1:M}$, respectively.

\textbf{Embedding of objects.}
To measure the relationship between visual entities more effectively, we incorporate depth information into object features. For the object $a^{obj}_n$, we denote its 2D geometry coordinate by $b^{obj}_n = [x^{tl}/W; y^{tl}/H; x^{br}/W; y^{br}/H]$, where $x^{tl}$ and $y^{tl}$ represent the coordinates of the upper left corner of object $a^{obj}_n$, $x^{br}$ and $y^{br}$ represent the coordinates of the bottom right corner of object $a^{obj}_n$, respectively. Then we denote the 3D spatial location feature of $a^{obj}_n$ by $x^{os}_n = [b^{obj}_n; dv^{obj}_n/255]$. The embedding of object $a^{obj}_n$ is calculated as
\begin{equation}
    x^{obj}_n = LN(W_{of}x^{of}_n) + LN(W_{s}x^{os}_n),
\end{equation}
where $W_{fr} \in \mathbb{R}^{t\times{d}}$ and $W_{s} \in \mathbb{R}^{t\times{5}}$, $t$ represents the~dimension of common space in multimodal transformer, and $LN$ represents layer normalization.

\begin{figure}[t]
  \centering

   \includegraphics[width=0.9\linewidth]{./depth_value.pdf}
   \caption{Illustrations for the process of getting the depth values and 3D geometric relationship modeling. The pie chart represents the gray value distribution of object $a^{obj}_n$. We take the gray values with the highest proportion as the depth value (0 means nearest) of object $a^{obj}_n$ and OCR token $a^{ocr}_m$. $b^{obj}_n$ and $b^{ocr}_m$ denote bounding boxes of $a^{obj}_n$ and $a^{ocr}_m$, respectively. With depth information, relationships between visual entities become more clear. }
   \label{fig:depth}
\end{figure}

\textbf{Embedding of OCR tokens.}
Unlike objects, OCR tokens not only contain rich visual clues but also contain rich semantic information. To get rich representations of OCR tokens, following Hu \textit{et al.}\cite{hu2020iterative} and Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we apply FastText\cite{bojanowski2017enriching} to extract the 300-dim sub-word feature $x^{ft}_m$ for OCR token $a^{ocr}_m$, and employ PHOC (Pyramidal Histogram of Characters)\cite{almazan2014word} to extract 604-dim character-level feature $x^{ph}_m$ for $a^{ocr}_m$. Similarly, the spatial location of $a^{ocr}_m$ is denoted as $x^{ts}_m = [b^{ocr}_m; dv^{ocr}_m/255]$, where $b^{ocr}_m = [x^{tl}/W; y^{tl}/H; x^{br}/W; y^{br}/H]$. To better fuse the three-dimensional relative spatial relationships into OCR features, we propose a Depth-enhanced Feature Updating Module (DeFUM, cf. \cref{subsec c}). In the real world, scene texts are attached to the surface of objects. We consider that visual features of objects can help model the visual relationship of OCR tokens. For example, if several different OCR tokens are on the same object, then these OCR tokens are likely to have a strong correlation. Therefore we fed the OCR token appearance features $x^{tf}_{1:M}$ into DeFUM along with object appearance features $x^{of}_{1:N}$, we can get the depth-enhanced OCR appearance feature $x^{tf'}_{1:M}$:
\begin{equation}
    x^{tf'}_{1:M} = DeFUM(x^{of}_{1:N},x^{tf}_{1:M},dv^{obj}_{1:N},dv^{ocr}_{1:M}),
\end{equation}
where $dv^{obj}_{1:N} \in \mathbb{R}^{N\times{1}}$ and $dv^{ocr}_{1:M} \in \mathbb{R}^{M\times{1}}$, respectively.
Considering the gap between visual features and semantic features, we do not update FastText and PHOC features (semantic embeddings).
Then we utilize the Semantic-guided Alignment Module (SgAM, cf. \cref{subsec e}) to integrate aligned visual concept information into the semantic representation of relevant OCR tokens
\begin{equation}
    x^{ft'}_{1:M} = SgAM(FT(a^{voc}_{1:K}),x^{ft}_{1:M}),
\end{equation}
where $x^{ft'}_{1:M}$ and $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, $FT(a^{voc}_k)$ represent the FastText embedding of visual concept $a^{voc}_k$ (cf. \cref{subsec d}).

Following Wang \textit{et al.}\cite{wang2021confidence}, we adopt confidence features of detected OCR tokens $\{x^{conf}_m\}_{m=1:M}$ due to some OCR tokens may be amiss. The final embedding of OCR token $a^{ocr}_m$ could be calculated by
\begin{equation}
\begin{aligned}
    x^{ocr}_m = LN(W_{tf}x^{tf'}_m + W_{ft}x^{ft'}_m + W_{ph}x^{ph}_m)\\ + LN(W_{s}x^{ts}_m) + LN(W_{conf}x^{conf}_m),
\end{aligned}
\end{equation}
where $W_{tf} \in \mathbb{R}^{t\times{d}}$, $W_{ft} \in \mathbb{R}^{t\times{300}}$,  $W_{ph} \in \mathbb{R}^{t\times{604}}$, $W_{s} \in \mathbb{R}^{t\times{5}}$ and $W_{conf} \in \mathbb{R}^{t\times{1}}$ are learnable parameters, and $LN$ represents layer normalization.

\begin{figure}[t]
  \centering

   \includegraphics[width=1.0\linewidth]{./dem.pdf}
   \caption{Illustration of the Depth-enhanced Feature Updating Module (DeFUM), which consists of a depth-aware self-attention module and $N$ layers of transformer encoder. The DeFUM could effectively enhance the appearance features of OCR tokens with the help of depth information, which is conducive to constructing the three-dimensional geometric relationships and improving the accuracy of captions.}
   \label{fig:dem}
\end{figure}
\subsection{Depth-enhanced Feature Updating Module}
\label{subsec c}
Unlike other multimodal tasks, such as VQA, which require high-quality relationships between objects, text-based image captioning requires models with stronger scene text relationship modeling capabilities. Therefore the quality of the OCR token features is more critical to the generated captions. For this reason, we propose the DeFUM to enhance the OCR appearance features $\{x^{tf}_m\}_{m=1:M}$ with the help of depth information. Intuitively, we consider that object features can play the role of a bridge linking its adjacent scene text. Therefore, we first obtain appearance embedding $x^v$ of visual entities by concatenating the visual features $\{x^{of}_n\}_{n=1:N}$ of objects $\{a^{obj}_n\}_{n=1:N}$ and the visual features $\{x^{tf}_m\}_{m=1:M}$ of OCR tokens $\{a^{ocr}_m\}_{m=1:M}$:
\begin{equation}
    x^v = Concat(x^{of}_{1:N}, x^{tf}_{1:M}),
\end{equation}
where $x^v \in \mathbb{R}^{(m+n)\times{d}}$. Subsequently, we calculate the query $Q$, key $K$, and value $V$ as follows:
\begin{equation}
    \left\{
    \begin{aligned}
    Q &= x^{v}W_Q \\
    K &= x^{v}W_K,\\
    V &= x^{v}W_V \\
    \end{aligned}
    \right.
\end{equation}
where query $Q \in \mathbb{R}^{(m+n)\times{d}}$, key $K \in \mathbb{R}^{(m+n)\times{d}}$, value $V \in \mathbb{R}^{(m+n)\times{d}}$. Then, we can obtain the attention weights matrix $A$ by:
\begin{equation}
    A = \frac{QK^T}{\sqrt{d}},
\end{equation}
where $K \in \mathbb{R}^{(m+n)\times{(m+n)}}$. To measure the relative depth between different visual entities, we calculate the relative depth score of $x^v_i$ and $x^v_j$ by
\begin{equation}
    R_{i,j} = log(\frac{dv_j}{dv_i}),
\end{equation}
where $dv = Concat(dv^{obj},dv^{ocr})$ and the relative depth map $R\in \mathbb{R}^{(m+n)\times{(m+n)}}$. Then the output of Depth-aware Self-Attention module (cf. \cref{fig:dem}) is updated by
\begin{equation}
    x^{vti} = LN(x^v + softmax(A + R)V),
\end{equation}
where the depth-aware visual feature $x^{vti} \in \mathbb{R}^{(m+n)\times{d}}$, $LN$ denotes layer normalization. Eventually, $x^{vti}$ is fed into $N$ layers of transformer\cite{vaswani2017attention} to update appearance features with depth information
\begin{equation}
    x^{vto} = transformer(x^{vti}),
\end{equation}
where the depth-enhanced appearance feature $x^{vto}\in \mathbb{R}^{(m+n)\times{d}}$ ($x^{vto} = [x^{of'}, x^{tf'}]$), $x^{of'}\in\mathbb{R}^{n\times{d}}$, $x^{tf'}\in \mathbb{R}^{m\times{d}}$, respectively.

\subsection{Salient Visual Object Concepts Extractor}
\label{subsec d}
To improve the modeling ability of the multimodal transformer to the scene objects, avoid the lack of vital objects, and generate more accurate captions, we propose a visual object concepts extractor to introduce semantic information of visual object concepts. Specifically, we employ a pre-trained model CLIP\cite{radford2021learning} to match $15$ object concepts related to the given image from vocabulary. Then we sort them by similarity score and take the top-K as visual object concepts. Visual object concepts and similarity score are represented as $\{a^{voc}_k\}_{k=1:K}$ and $\{a^{score}_k\}_{k=1:K}$. The embedding of visual object concepts are calculated as follows:
\begin{equation}
     x^{voc}_k = LN(W_{voc}FT(a^{voc}_k))\\ + LN(W_{score}a^{score}_k),
\end{equation}
where $FT(a^{voc}_k)$ means the FastText\cite{bojanowski2017enriching} embedding of $a^{voc}_k$, $W_{voc} \in \mathbb{R}^{t\times{300}}$ and $W_{score} \in \mathbb{R}^{t\times{1}}$.

\begin{figure}[t]
  \centering

   \includegraphics[width=1.0\linewidth]{./sgam.pdf}
   \caption{Illustration of the Semantic-guided Alignment Module (SgAM), which utilizes semantic attention to transfer the information of visual concepts to semantically aligned OCR Tokens' sub-word embedding. This operation is capable of making model emphasize visual concepts that are highly relevant to the scene text. Besides, SgAM helps multimodal transformer roughly model the spatial location of semantically aligned visual concepts.}
   \label{fig:sgam}
\end{figure}
\subsection{Semantic-guided Alignment Module}
\label{subsec e}
CLIP\cite{radford2021learning} is a large-scale pre-trained model based on contrastive learning, whose powerful representation ability has been fully demonstrated by the application of downstream tasks\cite{wu2022wav2clip, zhang2022pointclip}. However, CLIP embeds the whole image and sentence into textual and visual representations, which makes it difficult for CLIP's image encoder to capture fine-grained features\cite{ma2022x}. Therefore, it is hard to ground the area depicted by these visual concepts, which may prevent the model from effectively understanding the representations of visual concepts. In addition, to avoid the introduction of out-of-domain words, we do not use words that are not included in the vocabulary provided by M4C-Captioner\cite{sidorov2020textcaps} for visual object concept extraction. Accordingly, a small part of visual concepts may be invalid, because there may be less than $K$ words related to the given image in the limited vocabulary.

To alleviate the above potential problems, we propose a Semantic-guided Alignment Module (SgAM, cf. \cref{fig:sgam}). The sub-word embedding extracting model FastText is trained on Wikipedia data. Therefore, related words (e.g., Van and Vehicle) have relatively high semantic similarity. Taking advantage of this factor, we integrate semantic information of visual concepts into related OCR tokens features. The semantic integration operation not only helps the model distinguish visual concepts related to OCR tokens but also roughly models the spatial position of some visual concepts during the interaction with spatial position information of OCR tokens in the multimodal transformer.

Concretely, we first convert FastText embedding of visual concepts $FT(a^{voc}_{1:K})$ to query $Q_S$ and convert FastText embedding of OCR tokens $x^{ft}_{1:M}$ to value $K_S$, respectively:
\begin{equation}
    \left\{
    \begin{aligned}
    Q_S &= FT(a^{voc}_{1:K})W_{Q_S} \\
    K_S &= x^{ft}_{1:M}W_{K_S}, \\
    \end{aligned}
    \right.
\end{equation}
where $FT(a^{voc}_{1:K})\in \mathbb{R}^{K\times{300}}$, $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, while $W_{Q_S}$ and $W_{K_S}\in \mathbb{R}^{300\times{300}}$.

Subsequently, we adopt semantic attention to interact semantic information of visual concepts and OCR tokens
\begin{equation}
     Q'_S = softmax(\frac{Q_SK_S^T}{\sqrt{d_s}})^TFT(a^{voc}_{1:K}),
\end{equation}
where $Q'_S\in \mathbb{R}^{M\times{300}}$. Then we can get the final semantic embedding $x^{ft'}_{1:M}$ of OCR tokens
\begin{equation}
     x^{ft'}_{1:M} = L2Norm(x^{ft}_{1:M} + Q'_S),
\end{equation}
where $x^{ft'}_{1:M}$ and $x^{ft}_{1:M}\in \mathbb{R}^{M\times{300}}$, while $L2Norm$ represents L2 Normalization.

\subsection{Reasoning and Generation Module}
Following Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we adopt a multimodal transformer to encode all the obtained embeddings. Considering most OCR tokens (e.g., ``coca'' and ``spinner'' in \cref{fig:intro}) are not common words, therefore it is not appropriate to generate captions based on a fixed vocabulary. Following M4C-Captioner\cite{sidorov2020textcaps}, we utilize two classifiers for common vocabulary and candidate OCR tokens separately to generate words. Overall, multimodal transformer models multimodal embedding and generates common words $y'_t$. Afterward, a dynamic pointer network\cite{vinyals2015pointer} is employed to generate final words $y_t$:
\begin{equation}
\begin{aligned}
     y'_t, x^{tocr}_{1:M} = mmt(x^{obj}_{1:N}, x^{ocr}_{1:M}, x^{voc}_{1:K}, x^{dec}_{t-1}),
\end{aligned}
\end{equation}
where $mmt$ represents multimodal transformer, $x^{tocr}_m$ indicates the feature of $x^{ocr}_m$ updated by $mmt$, and $x^{dec}_{t-1}$ is the embedding of previous output $y'_{t-1}$, respectively. Eventually, the dynamic pointer network makes the final prediction from the common word in the fixed vocabulary and the special words $x^{tocr}_{1:m}$ from OCR tokens:
\begin{equation}
     y_t = argmax([l_{m}^{w}(y'_t),PN(y'_t, x^{tocr}_{1:m})]),
\end{equation}
where $l_{m}^{w}(\cdot)$ denotes a linear classifier for fixed common vocabulary, $PN(\cdot)$ denotes pointer network, and the caption $C$ consists of predicted words ${y_0, ..., y_t}$. We train our model by optimizing the Cross-Entropy loss:
\begin{equation}
     L(\theta) = -\sum_{t=1}^{T}log(\hat{y}_t|y_{1:t-1}, x^{tocr}_{1:m}),
\end{equation}
where $\hat{y}_t$ is the corresponding token in the ground truth.

\section{Experiments}
\label{sec4}

\subsection{Dataset and Settings}
\textbf{Dataset and evaluation metrics.}
The TextCaps dataset\cite{sidorov2020textcaps} is constructed for text-based image captioning, which contains 28408 images with 5 captions per image.  The training, validation, and test set contain 21953, 3166, and 3289 images, respectively. For evaluation metrics, we choose five widely used evaluation metrics for image captioning, i.e., BLEU-4\cite{papineni2002bleu}, ROUGE-L\cite{liao2017textboxes}, METEOR\cite{banerjee2005meteor}, SPICE\cite{anderson2016spice}, and CIDEr-D\cite{vedantam2015cider}. Following Sidorov \textit{et al.}\cite{sidorov2020textcaps}, we focus on the CIDEr-D metric when comparing different methods, because CIDEr-D has a high correlation with human evaluation scores and puts more weight on informative special tokens (e.g., OCR tokens) in the captions.

\begin{table*}
  \centering
    \caption{Comparison of our performance with other baseline models on the TextCaps validation set. TAP$^{\dagger}$ and ConCap$^{\dagger}$ are pre-trained models. Our DEVICE outperforms state-of-the-art models on METOR, ROUGE-L, SPICE, and CIDEr-D.}
    \label{tab1}
  \resizebox{1.5\columnwidth}{!}{
  \begin{tabular}{l|ccccc}
    \hline
    Model & BLEU-4 & METEOR & ROUGE-L & SPICE & CIDEr-D \\
    \hline
    \hline
    Up-Down (CVPR 2018)\cite{anderson2018bottom} & 20.1 & 17.8 & 42.9 & 11.7 & 41.9 \\
    AoA (ICCV 2019)\cite{huang2019attention} & 20.4 & 18.9 & 42.9 & 13.2 & 42.7 \\
    M4C-Captioner (ECCV 2020)\cite{sidorov2020textcaps} & 23.3 & 22.0 & 46.2 & 15.6 & 89.6 \\
    MMA-SR (ACM MM 2020)\cite{wang2020multimodal} & 24.6 & 23.0 & 47.3 & 16.2 & 98.0\\
    SS-Baseline (AAAI 2021)\cite{zhu2021simple} & 24.9 & 22.7 & 47.2 & 15.7 & 98.8\\
    CNMT (AAAI 2021)\cite{wang2021confidence} & 24.8 & 23.0 & 47.1 & 16.3 & 101.7\\
    ACGs-Captioner (CVPR 2021)\cite{xu2021towards} & 24.7 & 22.5 & 47.1 & 15.9 & 95.5\\
    MAGIC (AAAI 2022)\cite{zhang2022magic} & 22.2 & 20.5 & 42.3 & 13.8 & 76.6\\
    OMOT (ICMR 2022)\cite{tang2022ocr} & 26.4 & 22.2 & 47.5 & 14.9 & 95.1\\
    LSTM-R (CVPR 2021)\cite{wang2021improving} & \textbf{27.9} & 23.7 & 49.1 & 16.6 & 109.3\\
    \hline
    DEVICE (Ours) & 27.6 & \textbf{24.6} & \textbf{49.3} & \textbf{17.7} & \textbf{117.1}\\
    \hline
    M4C-Captioner(w/ GT OCRs)\cite{sidorov2020textcaps} & \textcolor[rgb]{0.5,0.5,0.5}{26.0} & \textcolor[rgb]{0.5,0.5,0.5}{23.2} & \textcolor[rgb]{0.5,0.5,0.5}{47.8} & \textcolor[rgb]{0.5,0.5,0.5}{16.2} & \textcolor[rgb]{0.5,0.5,0.5}{104.3}\\
    TAP$^{\dagger}$ (CVPR 2021)\cite{yang2021tap} & 25.8 & 23.8 & 47.9 & 17.1 & 109.2\\
    ConCap$^{\dagger}$ (AAAI 2023)\cite{wang2022controllable} & 31.3 & - & - & - & 116.7\\
    \hline
  \end{tabular}
  }
\end{table*}

\begin{table*}
  \centering
    \caption{Comparison of our performance with other baseline models on the TextCaps test set. Our DEVICE outperforms state-of-the-art models on all metrics, notably boosting CIDEr-D from 100.8 to 110.0.}
    \label{tab2}
  \resizebox{1.5\columnwidth}{!}{
  \begin{tabular}{l|ccccc}
    \hline
    Model & BLEU-4 & METEOR & ROUGE-L & SPICE & CIDEr-D \\
    \hline
    \hline
    Up-Down (CVPR 2018)\cite{anderson2018bottom} & 14.9 & 15.2 & 39.9 & 8.8 & 33.8 \\
    AoA (ICCV 2019)\cite{huang2019attention} & 15.9 & 16.6 & 40.4 & 10.5 & 34.6 \\
    M4C-Captioner (ECCV 2020)\cite{sidorov2020textcaps} & 18.9 & 19.8 & 43.2 & 12.8 & 81.0 \\
    MMA-SR (ACM MM 2020)\cite{wang2020multimodal} & 19.8 & 20.6 & 44.0 & 13.2 & 88.0\\
    SS-Baseline (AAAI 2021)\cite{zhu2021simple} & 20.2 & 20.3 & 44.2 & 12.8 & 89.6\\
    CNMT (AAAI 2021)\cite{wang2021confidence} & 20.0 & 20.8 & 44.4 & 13.4 & 93.0\\
    ACGs-Captioner (CVPR 2021)\cite{xu2021towards} & 20.7 & 20.7 & 44.6 & 13.4 & 87.4\\
    OMOT (ICMR 2022)\cite{tang2022ocr} & 21.2 & 19.6 & 44.4 & 12.1 & 84.9\\
    LSTM-R (CVPR 2021)\cite{wang2021improving} & 22.9 & 21.3 & 46.1 & 13.8 & 100.8\\
    \hline
    DEVICE (Ours) & \textbf{23.1} & \textbf{22.5} & \textbf{46.7} & \textbf{15.0} & \textbf{110.0}\\
    \hline
    M4C-Captioner(w/ GT OCRs)\cite{sidorov2020textcaps} & \textcolor[rgb]{0.5,0.5,0.5}{21.3} & \textcolor[rgb]{0.5,0.5,0.5}{21.1} & \textcolor[rgb]{0.5,0.5,0.5}{45.0} & \textcolor[rgb]{0.5,0.5,0.5}{13.5} & \textcolor[rgb]{0.5,0.5,0.5}{97.2}\\
    TAP$^{\dagger}$ (CVPR 2021)\cite{yang2021tap} & 21.9 & 21.8 & 45.6 & 14.6 & 103.2\\
    ConCap$^{\dagger}$ (AAAI 2023)\cite{wang2022controllable} & 27.4 & - & - & - & 105.6\\
    Human\cite{sidorov2020textcaps} & 24.4 & 26.1 & 47.0 & 18.8 & 125.5\\
    \hline
  \end{tabular}
  }
\end{table*}

\textbf{Settings and implementation details.}
To improve the robustness of the OCR system and increase the accuracy of detected OCR tokens, we employ the Rosetta OCR system\cite{borisyuk2018rosetta} and Google OCR system\cite{Googleocr} following Wang \textit{et al.}\cite{wang2021improving}. The number of OCR tokens for each image is limited to 80 at most. We extract 100 object appearance features for each picture, the dimension $d$ of appearance features is 2048, and the dimension $t$ of common embedding space in the multimodal transformer is 768. The maximum generation length is 30. DeFUM contains two layers of transformer encoder in our experiments, we set the layer number of the multimodal transformer to 4 and the number of self-attention heads to 12. We adopt default settings for other parameters following BERT-BASE\cite{devlin2018bert}. The fixed common vocabulary has 6736 words. To get visual object concepts, we filter top-$k$ ($k=5$ in our implement) visual concepts from the vocabulary by CLIP\cite{radford2021learning}. We retain the similarity score given by CLIP as a reference for measuring the relevance of visual concepts to the image. To get the confidence of OCR tokens, for those OCR tokens detected by Rosetta, we adopted the confidence provided by Wang \textit{et al.}\cite{wang2021confidence}. For those OCR tokens that only appear in one OCR system, we set the confidence to $0.9$.

We train the model for about 5 epochs (9000 iterations) on a single 3090 Ti GPU, and the batch size is 64. We adapt Adam\cite{kingad2015} optimizer, the initial learning rate is $1e$-$4$ and is declined to 0.1 times at 7000 iterations. We monitored the CIDEr-D metric to choose the best model and evaluate it on both the validation set and test set. All the experimental results are computed by Eval AI online platform submissions.

\subsection{Experimental Results}
\textbf{Compared models.}
We make comparisons with other models in \cref{tab1} and \cref{tab2}. \textbf{M4C-Captioner}\cite{sidorov2020textcaps}, \textbf{SS-Baseline}\cite{zhu2021simple}, \textbf{CNMT}\cite{wang2021confidence}, and \textbf{ACGs-Captioner}\cite{xu2021towards} are Transformer-based strong baseline models. \textbf{MMA-SR}\cite{wang2020multimodal}, \textbf{MAGIC}\cite{zhang2022magic}, \textbf{OMOT}\cite{tang2022ocr}, and \textbf{LSTM-R}\cite{wang2021improving} are LSTM-based strong baseline models.
\begin{itemize}
 \item \textbf{Up-Down}\cite{anderson2018bottom} and \textbf{AOA}\cite{huang2019attention} are widely used image captioning models. Both of them are trained and inferred without OCR tokens.
 \item \textbf{M4C-Captioner}\cite{sidorov2020textcaps} is a Transformer-based strong baseline model, which firstly introduces the OCR system into the TextCaps dataset.
 \item \textbf{SS-Baseline}\cite{zhu2021simple} introduces attention branches for visual and linguistic features of scene text.
 \item \textbf{CNMT}\cite{wang2021confidence} is the state-of-the-art Transformer-based model, which adopts the confidence of scene text, and introduces a non-repetitive generation paradigm.
 \item \textbf{ACGs-Captioner}\cite{xu2021towards} introduces the anchor-centred graph (ACG), and based on different ACGs generate diverse captions.
 \item \textbf{MAGIC}\cite{zhang2022magic} constructs multiple multimodal relational graphs to improve the diversity of captions and employs a cascaded generative adversarial network to generate unpaired captions.
 \item \textbf{MMA-SR}\cite{wang2020multimodal} is the first to explore sufficient two-dimensional spatial relationships between text tokens, and the model performance is remarkable.
 \item \textbf{OMOT}\cite{tang2022ocr} explores the master object for each OCR token, then builds scene graphs based on these master objects.
 \item \textbf{LSTM-R}\cite{wang2021improving} is the state-of-the-art LSTM-based model, which expands the two-dimensional geometric relationship measurement of OCR tokens on the basis of MMA-SR, such as IOU. LSTM-R can select high-quality OCR tokens with the help of a relation-aware pointer network.
 \item \textbf{TAP$^{\pmb{\dagger}}$}\cite{yang2021tap} is a text-aware pre-trained model, which is trained on TextVQA\cite{singh2019towards}, STVQA\cite{biten2019scene}, TextCaps\cite{sidorov2020textcaps}, and OCR-CC\cite{yang2021tap}.
 \item \textbf{ConCap$^{\pmb{\dagger}}$}\cite{wang2022controllable} is a pre-trained image captioning framework, which caption style can be controlled by prompts. ConCap$^{\dagger}$ is pre-trained on  Visual Genome\cite{krishna2017visual},  COCO\cite{lin2014microsoft}, Conceptual Caption\cite{sharma2018conceptual}, SBU Captions\cite{ordonez2011im2text}, and a filtered version of LAION\cite{schuhmann2021laion}.
 \item \textbf{M4C-Captioner (w/ GT OCRs)}\cite{sidorov2020textcaps} indicates \textbf{M4C-Cpationer} takes ground truth OCR tokens as input and evaluates on a subset of the TextCaps test set.
 \item \textbf{Human}\cite{sidorov2020textcaps} represents human-generated captions.
\end{itemize}

\begin{figure*}[t]
  \centering

   \includegraphics[width=1.0\linewidth]{./case_study3.pdf}
   \caption{Example of captions generated by strong baseline CNMT\cite{wang2021confidence}, DEVICE, and GT (ground truth). \textcolor{red}{Red} indicates inappropriate sence text generated by CNMT. The depth maps and visual object concepts are displayed in the middle. The score of visual object concepts measures how similar these concepts are to the image. \textcolor[rgb]{0,0.76,0.98}{Blue} indicates scene text and \textcolor[rgb]{0,0.69,0.23}{Green} indicates visual object concepts.}
   \label{fig:case}
\end{figure*}

\textbf{Main results on the TextCaps validation set.}
The comparisons on the validation set between our DEVICE and other models are shown in \cref{tab1}. Traditional methods UP-Down and AOA perform less well than other models with OCR tokens due to they cannot generate scene text. CNMT proves the effectiveness of confidence embedding and repetition mask, which improves CIDEr-D from $89.6$ to $101.7$ compared with M4C-Captioner. LSTM-R constructs sufficient 2D spatial relations and boosts all metrics significantly. By incorporating depth information and semantic information of visual object concepts with transformer, compared with SOTA model LSTM-R\cite{wang2021improving}, our DEVICE improves the performance on METEOR, SPICE, CIDEr-D by $0.9$, $1.1$, and $7.8$, respectively. The performance of DEVICE on BLEU-4 ranks second only to LSTM-R (0.3 below). Note that LSTM-R has cleaned the training captions by removing the undetected OCR symbols, which increases BLEU-4 by 0.6\cite{wang2021improving}. For a fair comparison, we train our model with raw captions provided by TextCaps\cite{sidorov2020textcaps}, following the majority\cite{sidorov2020textcaps, wang2021confidence, zhu2021simple, yang2021tap}. Moreover, CIDEr-D and METEOR have the highest correlation with human scores\cite{sidorov2020textcaps}, which shows that DEVICE significantly outperforms other baseline models from a human perspective. The performance of our model on the TextCaps validation set demonstrates the effectiveness of our DEVICE.
\begin{table}[t]
  \centering
  \caption{Comparison of iterations and time of DEVICE and other baseline models required to converge to the optimal CIDER.}
  \label{tab4}
  \resizebox{1.\columnwidth}{!}{
  \begin{tabular}{l|cccc}
    \hline
    Model & Batch Size & Iterations/epochs & Time\\
    \hline
    \hline
    M4C-Captioner & 128 & 12000/14 & 11h\\
    CNMT & 128 & 12000/14 & 12h\\
    ACGs-Captioner & 128 & 12000/14 & 12h\\
    SS-Baseline & 128 & 15000/17 & 14h\\
    LSTM-R & 50 & -/30 & -\\
    \hline
    DEVICE(w/o SgAM) & 64 & 12000/7 & 11h\\
    DEVICE & 64 & 9000/5 & 8h\\
    \hline
    TAP$^{\dagger}$ & - & 480000/- & -\\
    ConCap$^{\dagger}$ & 2880 & -/37 & -\\
    \hline
  \end{tabular}
  }
\end{table}
\begin{table}[t]
  \centering
  \caption{Ablation of each module in DEVICE on the TextCaps validation set. Google denotes Google OCR system, DI denotes Depth Information, DeFUM represents the Depth-enhanced Feature Updating Module, VOC indicates semantic information of Visual Object Concepts, and SgAm represents Semantic-guided Alignment Module.}
  \label{tab3}
  \resizebox{1\columnwidth}{!}{
  \begin{tabular}{lccccc|cc}
    \hline
    & Google & DI & DeFUM & VOC & SgAM & BLEU-4 & CIDEr-D \\
    \hline
    \hline
    1 & & & & & & 24.5 &101.5 \\
    2 & \checkmark & & & & & 24.9 & 105.3 \\
    3 & \checkmark & \checkmark & & & & 25.6 & 109.8 \\
    4 & \checkmark & \checkmark & \checkmark & & & 26.1 & 113.9\\
    5 & \checkmark & & & \checkmark & & 26.3 & 112.4\\
    6 & \checkmark & & & \checkmark & \checkmark & 26.5 & 114.4\\
    7 & \checkmark & \checkmark & \checkmark & \checkmark & & 27.4 & 115.9\\
    8 & \checkmark & \checkmark & \checkmark & \checkmark & \checkmark & \textbf{27.6} & \textbf{117.1}\\
    \hline
  \end{tabular}
  }
\end{table}

\textbf{Main results on the TextCaps test set.}
Compared with LSTM-R, our model shows superiority on the TextCaps test set, which gains rises of $0.2$, $1.2$, $0.6$, $1.2$, $9.2$ on BLEU-4, METEOR, ROUGE-L, SPICE, CIDEr-D, respectively. Significantly, the inspiring improvement on the CIDEr-D shows that our DEVICE generates more accurate scene text. The results of m4c-captioner (w/ GT OCRs)\cite{sidorov2020textcaps} are evaluated on a subset of the TextCaps test set, excluding those samples without OCR annotations. DEVICE outperforms m4c-captioner(w/ GT OCRs) verifies that our DEVICE is capable of compensating the negative impact of the error of OCR systems to some extent. Encouragingly, our model outperforms the pre-trained model TAP$^{\dagger}$ on all metrics, demonstrating that DEVICE utilizes limited training data more efficiently. The BLEU-4 score of ConCap$^{\dagger}$ is 27.3, which shows that ConCap$^{\dagger}$ has learned human language ability well from the large-scale corpus. However, DEVICE outperforms ConCap$^{\dagger}$ on CIDEr-D, which shows that the scene text in the captions generated by DEVICE is of higher quality. Finally, the gap between machines and humans narrows on the text-based image captioning task.

\textbf{Analysis of model convergence speed.}
In our experiments, we find that the SgAM module can significantly improve the model's converge speed, and achieve better results with fewer iterations. In \cref{tab4}, we list different models' iteration numbers and training epochs when achieving the optimal CIDEr-D. With SgAM, the required training time for DEVICE is reduced from 11 hours to 8 hours. This proves that semantic attention can help the multimodal transformer quickly select effective visual entities. M4C-Captioner, CNMT, ACGs-Captioner, and SS-Baseline are trained on 2 RTX 3090 Ti GPUs in \cref{tab4}.

\begin{table}[t]
  \centering
  \caption{Analysis of visual object concepts number $K$ on validation set. VOC indicates Visual Object Concepts.}
  \label{tab5}
  \resizebox{.85\columnwidth}{!}{
  \begin{tabular}{lc|ccc}
    \hline
    Model & Number of VOC & CIDEr-D \\
    \hline
    \hline
    DEVICE & $K$ = 3 & 115.9 & \textcolor{red}{$\downarrow$}1.2\\
    DEVICE & $K$ = 5 & \textbf{117.1} &\\
    DEVICE & $K$ = 8 & 116.5 & \textcolor{red}{$\downarrow$}0.6\\
    DEVICE & $K$ = 10 & 116.1 & \textcolor{red}{$\downarrow$}1.0\\
    \hline
  \end{tabular}
  }
\end{table}

\subsection{Ablation Studies}
To analyze the respective effects of each module on the model performance, we conduct ablation experiments, as shown in \cref{tab3}. Comparing Line 1 and Line 2, CIDEr-D is boosted from $101.5$ to $105.3$. It is evident that more affluent and more accurate OCR tokens positively impact the model performance. While the improvement on BLEU-4 is relatively moderate, we think one reason may be that BLEU-4 treats all matched words equally, but CIDEr-D pays more attention to OCR tokens. Line $2$ and $3$ prove that simply concatenating depth values with two-dimensional space coordinates boosts the performance substantially, which increases BLEU-4 by $0.7$ and CIDEr-D by $4.5$, respectively. Comparing Line $3$ and $4$, we find that updating the appearance features of OCR tokens under the supervision of depth information boosts all metrics. The significant improvements between Line $4$ and $7$ demonstrate the effectiveness of visual object concepts' semantic information, which boosts BLEU-4 by $1.3$ and CIDEr-D by $2.0$, respectively. The introduction of SgAM reduces the iterations for the model to converge to the optimum, and improves the model's performance on CIDEr-D by $2.0$, respectively. More remarkably, the total increase ratio of CIDEr-D is $11.8$ and BLEU-4 is $2.7$. This demonstrates that all modules work together very efficiently.

We also evaluate the influences of the number $K$ of visual object concepts on the TextCaps validation (cf. \cref{tab5}). The experimental results indicate that some salient visual object concepts may be missed when $K$ is less than~5. Meanwhile, irrelevant visual concepts may be retrieved when $K$ is larger than 7. Therefore, we set $K$ to $5$.

\subsection{Qualitative Analysis}
We show some cases which are generated by CNMT\cite{wang2021confidence} and DEVICE on the TextCaps validation set in \cref{fig:case},  where ``GT'' means ground truth. We pick CNMT\cite{wang2021confidence} for comparison because our geometric relationship construction methods are the same except for depth, and both are Transformer-based structures. Case 1 and Case 2 demonstrate the effectiveness of visual entities' 3D spatial relationship modeling ability. Case 3 and 4 illustrate the influences of effectively utilize visual object concepts by SgAM. In Case 1, CNMT mistakenly takes ``ats'' and ``mobile'' as tokens in the same plane and connects them incorrectly. By introducing 3D geometric relationship, our DEVICE generates accurate scene text. In Case 2, CNMT incompletely represents the text in the sign and confuses the positional relationship between the airplane and the text on the sign. DEVICE enhances the correlation of scene text on the warning sign and clarifies the spatial relationship between visual entities by introducing 3D geometric relationships. For Case 3, CNMT only generates relatively coarse-grained words such as ``many books''. In contrast, with the help of semantic information of visual object concepts and SgAM, DEVICE is capable of generating more accurate words ``comic books'', like human. In the last case, unlike CNMT, which ignores the visual object ``mascots'', DEVICE generates a more comprehensive description, enabling rational use of scene text ``pylon''. The aforementioned cases demonstrate the effectiveness of DEVICE well.

\begin{figure}[t]
  \centering

   \includegraphics[width=1.0\linewidth]{./case_study5.pdf}
   \caption{Negative samples of CNMT and DEVICE on the TextCaps validation set. The objects are inside the \textcolor[rgb]{0,0.69,0.23}{Green} bounding boxes, the OCR tokens are inside the \textcolor[rgb]{0,0.76,0.98}{Blue} bounding boxes. GT represents the Ground Truth.}
   \label{fig:case2}
\end{figure}

However, in Case 4, we can see the gap between our DEVICE and human, we consider one possible reason is that humans have common knowledge. Although adopting CLIP\cite{radford2021learning} to extract the visual object concept ``mascot'' in Case 4 can also be regarded as a way of implicitly using external knowledge, this knowledge is not fine-grained and accurate enough compared to ``Mario'' and ``Luigi''.

We also visualize some negative examples generated by our model on TextCaps validation dataset in \cref{fig:case2}. In Case a, strong baseline CNMT simply depicts the characters on the dial. In contrast, our model attempts to generate higher-level information, namely time. However, our model reads the time of the watch incorrectly. We consider the reason is that the ability of DEVICE to model the relationship between objects and scene text is still limited. Besides, it is difficult for models to read the time on the dial without numbers. In Case b, CNMT and DEVICE cannot correctly capture the information of ``measuring length'', which shows that current models are generally less capable of perceiving intent in scenes.

\section{Conclusion}

In this paper, we propose a DEpth and VIsual ConcEpts Aware Transformer (DEVICE) for TextCaps, which is capable of generating accurate and comprehensive captions for given images. We introduce depth information and design a depth-enhanced feature updating module to improve OCR appearance features, which is capable of facilitating the construction of 3D geometric relations. Meanwhile, we introduce the semantic information of salient visual object concepts and propose a semantic-guided alignment module to interact visual concepts with aligned  scene text, which improves the integrity of captions. More remarkably, our model achieves state-of-the-art performances on the TextCaps test set. By comparing experimental results with human annotations, we consider that extracting more explicit external knowledge can further improve this task.

{
\bibliographystyle{IEEEtran}


\section*{Acknowledgment}

Financial support from Science Foundation Ireland (SFI) grants 17/CDA/4760 (SoftEdge), 18/RI/5721 (OpenIreland) and 13/RC/2077\_p2 (CONNECT) is gratefully  acknowledged.

\bibliographystyle{IEEEtran}

\section{CPU utilization in NFV}
\label{sec:CPU_utilisation}

In this section we investigate the CPU utilization for two \acp{VNF}, \ac{SDN} and \ac{SDR}, through testbed measurements and data analysis.

\subsection{CPU utilization in SDN}

In order to characterize the CPU utilization of SDN functions we have investigated multiple bandwidths options in a linear topology. The measurements focus on the data plane, i.e. the CPU consumption of Open vSwitches (OVS switches), and hosts.

\begin{figure}[h]
	\centering
	   \vspace{+0.1in}
\includegraphics[width=\columnwidth]{figures/linear1030.png}
	\caption{Experimental set up with linear topology}
	\label{fig:linear}
    \vspace{-0.1in}
\end{figure}

Fig.~\ref{fig:linear} shows our experimental set up of \ac{SDN} CPU utilization measurement. We use the Mininet simulator~\cite{mininet} for SDN and two separate virtual machines (VMs) for hosting the control plane (SDN controller) and the data plane (OVS switches and hosts), respectively. The $htop$ command is used for monitoring the CPU utilization at the data plane. We use $iperf$ UDP data transmission tool integrated in the Mininet to simulate the data transmission. We measure the CPU utilization of the OVS switches and hosts for the data plane (in the Mininet virtual machine) in different data transmission rates. In the testing scenario of linear topology, we set up 5 OVS switches with one host attached to each switch, and the data transmission is from the host on the left end to the host on the right end, i.e. the route traverse 5 OVS switches. We use the Linux application $iperf$ to generate data transmission in Mininet. We use 2 CPU cores to host the data plane and use $htop$ program in Linux to measure the CPU utilization. The CPU model we use for SDN experiment is \textit{Intel Core i7-4810MQ@2.80GHz}. $htop$ is a Linux command to monitor system resource utilization \cite{htop}.
The results are shown in Fig. \ref{fig:SDN_utilization_boxplot}, which is a boxplot with data plots in red dots. For each data transmission rate we have measured 20 data points. As the transmission rate increases from 100kbps to 10Gbps (logarithmic scale), the CPU utilization appears to increase in a linear scale, with mean value increasing from 20\% to 175\%. Note that we are using virtual \ac{SDN} switches here implemented by virtual machines, thus the CPU utilization is affected by the CPU processing capacity. The overall CPU utilization should lower for the same data rate if we use more powerful CPUs.

\begin{figure}[h]
\vspace{-0.2in}
	\centering
\includegraphics[width=\columnwidth]{figures/CPU_SDN.png}
\vspace{-20mm}
\vspace{0.15in}
	\caption{SDN CPU utilization results}

	\label{fig:SDN_utilization_boxplot}
\end{figure}

\subsection{CPU utilisation in SDR}

Next, we measure the CPU utilization for an SDR with different parameter settings. We use srsRAN~\cite{srsRAN} to build a software LTE base station, whose RF runs in a USRP B210. The OpenIreland testbed is located at the CONNECT centre, in Trinity College Dublin (whose SDR part is shown in Fig.~\ref{fig:SDR_picture}).

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/testbed_picture.jpg}
	\caption{Testbed picture for SDR}
	\label{fig:SDR_picture}
    \vspace{0.1in}
\end{figure}

We have measured CPU utilization for scenarios involving use of different bandwidth and \acp{PRB}. Table~\ref{tab:PRB50} shows the results of 50 PRBs for the LTE eNodeB (our CPU is not powerful enough to carry out 100 PRB experiments without down sampling the wireless signals in the signal processing phase of \ac{SDR}). We have used the Linux application $iperf$ for both UDP and TCP packet transmission tests. The CPU utilization in the table is the mean value taken from 20 experimental data points. The results show that the CPU utilization increases when bandwidth settings for data transmission increases. The real throughput measured in the air is limited around 20 Mbps, and the CPU utilization is limited around 85\%-90\%. For this reason, also the packet loss for UDP and number of retries for TCP both start to increase when bandwidth setting increases beyond 20Mbps. Please note that we were using 16QAM as the modulation and coding schemes to carry out the experiments, therefore it is not able to reach the maximum throughput of 64QAM. The CPU model we use for SDR experiment is \textit{Intel NUC i7-8559U@2.70GHz}. Our CPU is not powerful enough for high modulation of 64QAM without down sampling the wireless signals in the signal processing phase of \ac{SDR}.

Table~\ref{tab:PRB25} shows the results for the case of 25 PRBs. 16 QAM is again used in this scenario and the real throughput caps at around 10 Mbps. The CPU utilization in the table is also the mean value taken from 20 experimental data points. We can observe that the CPU utilization caps at around 60\% - 65\% percent. This is lower compared to the 20Mbps throughput using 50 PRBs, which means that in this 25 PRB scenario the full potential of the CPU processing is not reached. The data rate and CPU utilization are both constrained by the lower PRB numbers and low order of modulation and coding schemes.

\begin{table}[h]
\centering
\caption{Testbed measurement results for SDR PRB 50}
\label{tab:PRB50}
\begin{adjustbox}{width=\columnwidth, center}
\begin{tabular}{|l|lll|lll|}
\hline
PRB 50                                                                & \multicolumn{3}{c|}{TCP}                                                                                & \multicolumn{3}{c|}{UDP}                                                                                \\ \hline
\begin{tabular}[c]{@{}l@{}}Bandwidth \\ setting\\ (Mbps)\end{tabular} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}Real\\ Throughput\\ (Mbps)\end{tabular}} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}CPU \\ utilization\\ (\%)\end{tabular}} & \begin{tabular}[c]{@{}l@{}}Number\\ of\\ Retries\end{tabular} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}Real \\ Throughput\\ (Mbps)\end{tabular}} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}CPU \\ utilization\\ (\%)\end{tabular}} & \begin{tabular}[c]{@{}l@{}}Packet\\ Loss\end{tabular} \\ \hline
1                                                                     & \multicolumn{1}{l|}{1}                                                                  & \multicolumn{1}{l|}{28.5}                                                              & 0                                                             & \multicolumn{1}{l|}{1}                                                                  & \multicolumn{1}{l|}{29.8}                                                              & 0                                                     \\ \hline
5                                                                     & \multicolumn{1}{l|}{5}                                                                  & \multicolumn{1}{l|}{68.3}                                                              & 0                                                             & \multicolumn{1}{l|}{5}                                                                  & \multicolumn{1}{l|}{49}                                                                & 0                                                     \\ \hline
10                                                                    & \multicolumn{1}{l|}{9.8}                                                                & \multicolumn{1}{l|}{73.4}                                                              & 0                                                             & \multicolumn{1}{l|}{10}                                                                 & \multicolumn{1}{l|}{62.8}                                                              & 0                                                     \\ \hline
15                                                                    & \multicolumn{1}{l|}{14.6}                                                               & \multicolumn{1}{l|}{76.3}                                                              & 0                                                             & \multicolumn{1}{l|}{15}                                                                 & \multicolumn{1}{l|}{74.2}                                                              & 0                                                     \\ \hline
20                                                                    & \multicolumn{1}{l|}{19.3}                                                               & \multicolumn{1}{l|}{87.1}                                                              & 0                                                             & \multicolumn{1}{l|}{20}                                                                 & \multicolumn{1}{l|}{84.7}                                                              & 0                                                     \\ \hline
25                                                                    & \multicolumn{1}{l|}{21.2}                                                               & \multicolumn{1}{l|}{90.3}                                                              & 36                                                            & \multicolumn{1}{l|}{20.6}                                                               & \multicolumn{1}{l|}{86.6}                                                              & 16                                                    \\ \hline
30                                                                    & \multicolumn{1}{l|}{21.2}                                                               & \multicolumn{1}{l|}{90.3}                                                              & 58                                                            & \multicolumn{1}{l|}{20.6}                                                               & \multicolumn{1}{l|}{86.1}                                                              & 30                                                    \\ \hline
35                                                                    & \multicolumn{1}{l|}{21.1}                                                               & \multicolumn{1}{l|}{90.8}                                                              & 44                                                            & \multicolumn{1}{l|}{20.3}                                                               & \multicolumn{1}{l|}{86}                                                                & 41                                                    \\ \hline
40                                                                    & \multicolumn{1}{l|}{21.1}                                                               & \multicolumn{1}{l|}{90.9}                                                              & 30                                                            & \multicolumn{1}{l|}{20.6}                                                               & \multicolumn{1}{l|}{86.6}                                                              & 48                                                    \\ \hline
45                                                                    & \multicolumn{1}{l|}{21.1}                                                               & \multicolumn{1}{l|}{90.9}                                                              & 31                                                            & \multicolumn{1}{l|}{20.1}                                                               & \multicolumn{1}{l|}{86}                                                                & 54                                                    \\ \hline
50                                                                    & \multicolumn{1}{l|}{21.2}                                                               & \multicolumn{1}{l|}{90.4}                                                              & 40                                                            & \multicolumn{1}{l|}{20.1}                                                               & \multicolumn{1}{l|}{85.1}                                                              & 59                                                    \\ \hline
100                                                                   & \multicolumn{1}{l|}{21.2}                                                               & \multicolumn{1}{l|}{90}                                                                & 50                                                            & \multicolumn{1}{l|}{19.5}                                                               & \multicolumn{1}{l|}{86.7}                                                              & 80                                                    \\ \hline
\end{tabular}
\end{adjustbox}
\end{table}

\begin{table}[]
\centering
\caption{Testbed measurement results for SDR PRB 25}
\label{tab:PRB25}
\begin{adjustbox}{width=\columnwidth, center}
\begin{tabular}{|l|lll|lll|}
\hline
PRB 25                                                                & \multicolumn{3}{c|}{TCP}                                                                                & \multicolumn{3}{c|}{UDP}                                                                                \\ \hline
\begin{tabular}[c]{@{}l@{}}Bandwidth \\ setting\\ (Mbps)\end{tabular} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}Real\\ Throughput\\ (Mbps)\end{tabular}} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}CPU \\ utilization\\ (\%)\end{tabular}} & \begin{tabular}[c]{@{}l@{}}Number\\ of\\ Retries\end{tabular} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}Real \\ Throughput\\ (Mbps)\end{tabular}} & \multicolumn{1}{l|}{\begin{tabular}[c]{@{}l@{}}CPU \\ utilization\\ (\%)\end{tabular}} & \begin{tabular}[c]{@{}l@{}}Packet\\ Loss\end{tabular} \\ \hline
1                                                                     & \multicolumn{1}{l|}{1}                                                                  & \multicolumn{1}{l|}{42.1}                                                              & 0                                                             & \multicolumn{1}{l|}{1}                                                                  & \multicolumn{1}{l|}{37.4}                                                              & 0                                                     \\ \hline
5                                                                     & \multicolumn{1}{l|}{4.99}                                                               & \multicolumn{1}{l|}{50.3}                                                              & 0                                                             & \multicolumn{1}{l|}{4.99}                                                               & \multicolumn{1}{l|}{51}                                                                & 0                                                     \\ \hline
10                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{61.1}                                                              & 54                                                            & \multicolumn{1}{l|}{9.13}                                                               & \multicolumn{1}{l|}{62.3}                                                              & 8.1                                                   \\ \hline
15                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{64.4}                                                              & 58                                                            & \multicolumn{1}{l|}{9.03}                                                               & \multicolumn{1}{l|}{60.1}                                                              & 39                                                    \\ \hline
20                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{63.1}                                                              & 57                                                            & \multicolumn{1}{l|}{8.92}                                                               & \multicolumn{1}{l|}{63.3}                                                              & 55                                                    \\ \hline
25                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{64.1}                                                              & 54                                                            & \multicolumn{1}{l|}{8.82}                                                               & \multicolumn{1}{l|}{63.1}                                                              & 64                                                    \\ \hline
30                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{63.7}                                                              & 58                                                            & \multicolumn{1}{l|}{8.74}                                                               & \multicolumn{1}{l|}{63.1}                                                              & 71                                                    \\ \hline
35                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{65}                                                                & 53                                                            & \multicolumn{1}{l|}{7.71}                                                               & \multicolumn{1}{l|}{63.3}                                                              & 78                                                    \\ \hline
40                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{61.7}                                                              & 59                                                            & \multicolumn{1}{l|}{7.48}                                                               & \multicolumn{1}{l|}{62.9}                                                              & 81                                                    \\ \hline
45                                                                    & \multicolumn{1}{l|}{9.47}                                                               & \multicolumn{1}{l|}{59.5}                                                              & 58                                                            & \multicolumn{1}{l|}{7.21}                                                               & \multicolumn{1}{l|}{62.1}                                                              & 84                                                    \\ \hline
50                                                                    & \multicolumn{1}{l|}{9.46}                                                               & \multicolumn{1}{l|}{63.6}                                                              & 58                                                            & \multicolumn{1}{l|}{6.54}                                                               & \multicolumn{1}{l|}{63.3}                                                              & 86                                                    \\ \hline
\end{tabular}
\end{adjustbox}
\end{table}

\section{Conclusion}
\label{sec:conclusion}

In this paper, we have used testbed measurements with real cloud-based virtual machines to analysis power consumption of \ac{NFV} processes, in particular, \ac{SDN} and \ac{SDR} functions. We have investigated the CPU utilization and parallelization for different configurations and scenarios and developed an energy saving scheme based on the data collected from the measurements. Our power saving methods can save up to 20\% power consumption compared to the case where \ac{SDN} and \ac{SDR} are deployed separately in the system.
\section{Introduction}
\label{sec:introduction}

\ac{NFV} technologies are being proposed to be used broadly in next-generation networks, especially in cloud-based network architecture, e.g. \ac{MEC}, \ac{IoT}, etc \cite{vDBA, Das_JOCN}. The softwarization and virtualization of the network functions enables the opportunity for cloud technologies being embedded in the \ac{NFV}.
\ac{SDR} and \ac{SDN}
play important roles in next generation wireless and wired networks, as together they can enable unprecedented flexibility in capacity allocation and service differentiation \cite{SDN_SDR_magazine}. However, these are conventionally deployed separately (i.e., in different servers), which can lead to suboptimal use of resources. In this paper we investigate the power consumption of the two technologies and propose power saving methodologies, based on the coexistence of the two paradigms on the same group of CPU cores.
Our previous work~\cite{ChZhIo-IoT20} proposed a power saving strategy based on reinforcement learning by dynamically adding and reducing CPU cores for \ac{SDN} and \ac{SDR}. However, in that work the parameter settings of the active-reward pairs are assumed with normalized values. In this paper, we go deeper into the power consumption and CPU utilization by carrying out extensive testbed measurements, collecting and analyzing data of a real cloud-based system for \ac{SDN} and \ac{SDR} in the Trinity College Dublin OpenIreland testbed~\cite{open_ireland}.

There are other works related to our work. For instance, authors in~\cite{AyGaCo21} presented an experimental study of power consumption of virtualized base stations (vBSs), investigating its relationship with performance. However, they do not investigate \ac{SDN} or power saving schemes. Authors in~\cite{EiKiMo19} proposed a dynamic resource allocation scheme in Software-Defined-Radio Access Networks based on statistical evaluations. However, they focus on allocation of radio resources rather than CPU computational resources. Authors in~\cite{IsMa2020} studied several types of functional splits in the \ac{NFV} of a dual-site network in Virtualized Radio Access Networks with \ac{SDR}. The paper investigated the combined optimization of the power consumption and the mid-haul bandwidth of the functional splits. However, there was no real testbed measurements involved in the paper.
Authors in \cite{FeCeOc16} solved an optimization problem to minimize the power consumption of \ac{SDN} by Integer Linear Programming and routing constraints, however there was no testbed measurement or experiment involved either.
Based on the investigation of related works, the contribution of our paper can be summarized as follows.

\begin{itemize}
    \item We propose an \ac{SDN} and \ac{SDR} integrated \ac{NFV} architecture that is a cloud-based virtualization architecture.
We investigate power consumption of virtual switches and \ac{SDR} by real testbed measurement.
    \item We propose a power optimization algorithm with \ac{SDN} and \ac{SDR} integrated CPU deployment, and compare it to the case where these two functions are deployed separately.
\end{itemize}

The results of our work show different CPU utilization features for \ac{SDN} and \ac{SDR} which enables the feasibility of power savings of an integrated CPU deployment of these two important \ac{NFV} functions. Our power saving scheme can save up to 20\% power consumption compared to the approach where \ac{SDN} and \ac{SDR} are deployed separately.

We propose an \ac{MEC} as an example for a use case of \ac{SDN} and \ac{SDR} integrated architecture, as shown in Figure~\ref{fig:MEC_architecture}. This architecture can be for example adopted in systems for inter-vehicle communications \cite{1543745}.
The architecture is all software-defined and network functions are virtualized. As the two major network functions for this architecture, \ac{SDN} and \ac{SDR} functions are integrated in the edge servers (i.e. \ac{MEC} nodes). Vehicular networks are illustrated at end user side. Inside the \ac{MEC} nodes, SDN switches, eNodeB \acp{BBU}, 4G/5G core function, and computational resources for \ac{MEC} tasks utilize the same group of servers. In this paper, we focus on the integration of the communication part, i.e., \ac{SDN} and \ac{SDR} components.

\begin{figure}[h]
	\centering
	\includegraphics[width=\columnwidth]{figures/overall_architecture.png}
	\caption{Proposed SDN/SDR integrated architecture with MEC as a use case example}
	\label{fig:MEC_architecture}
	\vspace{-0.1in}
\end{figure}

\section{Power Saving Methods}
\label{sec:power_savings}

In this section we investigate the power saving methods for the proposed \ac{MEC} and \ac{NFV} architecture by dynamically allocating different number of CPUs to \ac{SDN} and \ac{SDR} processes.

\subsection{Dynamic CPU allocation and process parallelisation}

In our cloud system, we propose to use the same group of CPU cores for \ac{SDN} and \ac{SDR}. We investigate the parallelization features by switching from one CPU core to two CPU cores for one process of \ac{SDN} or \ac{SDR}, using the Linux system command in $echo$ $1$ $>/sys/devices/system/cpu1/online$ and $echo$ $0$ $>/sys/devices/system/cpu1/online$ for adding/reducing CPU cores. We measure the CPU utilization for the same process running on one CPU core or on two CPU cores after one CPU is added to the virtual machine.

\subsubsection{\ac{SDN} testing}
We use Mininet to create an SDN-controlled experimental virtual network and use $iperf$ to transmit 500GB of data, with 10Gbps bandwidth in a single link on different CPU number systems.
We use Linux command $htop$ to measure the CPU utilization during our experiment.
We monitor the CPU utilization rate during the transmission. When we use two CPUs to process the same \ac{SDN} task, the task appears to be parallelized on the two CPUs, with a sample $htop$ screenshot shown in Fig. \ref{fig:htop_SDN}.
We collect more samples (100 data points measured) of the CPU utilization to draw the box plots of the samples in Fig. \ref{fig:boxplot_SDN}. The results show that \ac{SDN} data plane switching processes are usually multi-threaded and can be parallelized.

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/htop_SDN.png}
	\caption{CPU Utilization htop measurement for SDN}
	\label{fig:htop_SDN}
    \vspace{0.1in}
\end{figure}

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/Boxplot_SDN2.png}
	\caption{CPU Utilization rate for \ac{SDN}}
	\label{fig:boxplot_SDN}
    \vspace{0.1in}
\end{figure}

\subsubsection{\ac{SDR} testing}

In this subsection, we use srsRAN to create an experimental virtual network for \ac{SDR} and use $iperf$ to transmit with 10Mbps bandwidth. We monitor the CPU utilization rate during the transmission. When we use two CPUs to process the same \ac{SDR} task, the task does not appear to be parallelized on the two CPUs, with the $htop$ sample screenshot in Fig.~\ref{fig:htop_SDR}, and more samples in box plots shown in Fig. \ref{fig:boxplot_SDR}. Among these two CPUs, the higher utilized CPU has almost the same utilization from the one CPU case and the lower utilized CPU has very low utilization. The results show that \ac{SDR} processes are usually single-thread processes that can not be parallelized.

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/htop_SDR.png}
	\caption{CPU Utilization htop measurement for SDR}
	\label{fig:htop_SDR}
    \vspace{0.1in}
\end{figure}

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/Boxplot_SDR.png}
	\caption{CPU Utilization rate for \ac{SDR}}
	\label{fig:boxplot_SDR}
    \vspace{0.1in}
\end{figure}

\subsection{Shared CPU Utilization vs Separate CPU Utilization}

The algorithm we design for power savings in this paper is based on the aforementioned experimental results that the \ac{SDN} processes are multi-thread processes and easier to be parallelized, while \ac{SDR} processes are not. This gives an opportunity to optimize the total power consumption by packing the \ac{SDN} and \ac{SDR} processes in the same cluster of CPU cores. Assuming the function $P(x)$ is the function for the power consumption vs CPU utilization, the objective function is the following Equation (\ref{eqn:shared_sdn_sdr}):

\begin{equation}
    minimise: P_{total} = \sum_{c=1}^{C}P(U(c))
    \label{eqn:shared_sdn_sdr}
\end{equation}

where $c$ is the index of the CPU core in the $C$ number of cores. $U(c)$ is the CPU utilization percentage for the CPU core $c$ and $U(c)\in [0,100]$. In this equation we don't differentiate \ac{SDN} and \ac{SDR} CPU utilization, since they share the same group of CPU cores.

If \ac{SDN} and \ac{SDR} utilize the CPU cores separately, we have the total power consumption shown as the follow Equation(\ref{eqn:separate_sdn_sdr}):

\begin{equation}
    P_{total}=P(U_{SDN})+P(U_{SDR})
    \label{eqn:separate_sdn_sdr}
\end{equation}

In this equation, $U_{SDN}$ is the CPU utilization for \ac{SDN} and $U_{SDR}$ is the CPU utilization for \ac{SDR}. In this case the CPU utilization is decided by the processes that run separately for \ac{SDN} and \ac{SDR} without chances for optimization.

\subsection{Optimization Algorithm for Power Consumption}

We design the power consumption optimization algorithm for the \ac{SDN} and \ac{SDR} integrated system, as Algorithm \ref{alg:power_saving}.

The algorithm is an evolved version of the idea of best-fit algorithm. Since the \ac{SDR} processes are usually single thread and not capable to be parallelized, the \ac{SDR} CPU utilization comes in large chunks. On the contrary, the \ac{SDN} processes come in small chunks and can be parallelized. Therefore, the algorithm tries to fit first the large \ac{SDR} processes into the CPU cores, and then use the remaining space in CPU cores to accommodate the \ac{SDN} processes.

We have 3 usecase scenarios to test the performance of our power saving algorithm.

\begin{itemize}
    \item Usecase 1: a scenario where a large amount of \ac{SDR} bandwidth is needed but less \ac{SDN} data plane operations are needed (e.g., a local mobile network for content uploading/downloading to local servers, like mobile hotspot, Virtual Reality, Augmented Reality, etc.).
    \item Usecase 2: a scenario where little \ac{SDR} bandwidth is needed, but a large amount of \ac{SDN} data plane operations are needed, i.e., highly dynamic networking with several routing changes (e.g., mobile edge cloud and migration cases where migrations of computing tasks between edge servers are frequent, but wireless data transmission rate is low).
    \item Usecase 3: a scenario where both high \ac{SDR} bandwidth and large amount of \ac{SDN} operations are needed for both high bandwidth and high dynamicity (i.e., remote operations, mobile video streaming etc).

\end{itemize}
\begin{algorithm}
\caption{CPU power consumption optimization algorithm}
\label{alg:power_saving}
\begin{algorithmic}[1]
\State given $U^{i}_{SDR}, U^{j}_{SDN}$: the CPU utilization of each SDR process $i$ and SDN process $j$
\State group all SDN processes together to be $U_{SDN} = \sum_{j=1}^{J} U^{j}_{SDN}$.
\State $i \in I$ and $j \in J$
\For {all CPU core $c \in C$}
    \State Check the remaining CPU capacity of $c$: $1 - U(c)$
    \State Fit the largest SDR $U^{i}_{SDR}$ into $c$: $U^{i}_{SDR} \leq 1 - U(c)$
    \State Get the remaining CPU capacity: $\overline{U(c)} = 1 - U(c) - U^{i}_{SDR}$
    \State Fit the $\overline{U(c)}$ with a chunk of $U_{SDN}$ and calculate the remaining $ U_{SDN}$ by $U_{SDN} - U(c)$
    \Comment{Comment: because SDN processes can be fully parallelised according to our testbed measurement}
\EndFor

\end{algorithmic}
\end{algorithm}
Our power saving algorithm was implemented in Python and tested through extensive simulations.
Table~\ref{tab:usecases} shows the 3 usecase settings for the process computing loads, used as input for the \ac{SDN} data plane and \ac{SDR} simulations. Each use case differs with respect to number of processes that need to be allocated to \ac{SDN} and \ac{SDR}, and CPU utilization for each process. We assume CPU utilization is randomly distributed within a given range (shown in Table~\ref{tab:usecases}) with uniform distribution. Figure \ref{fig:simulation_procedure} shows the whole procedure of our simulation, including 3 main components: the stochastic task generator (for the 3 usecases mentioned above), the power saving optimisation algorithm, and the CPU utilisation and power consumption analysis (calculating number of CPU cores and utilization, as well as power consumption). Calculation of power consumption is baseed on values derived from CPU power analysis reports \cite{spec_power_asus} and \cite{spec_power_hp}, as described below.

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/algorithm_flow_chart.png}
	\caption{Simulation procedure}
	\label{fig:simulation_procedure}
\end{figure}

\begin{table}
\begin{center}
\caption{\label{tab:usecases}CPU process information for 3 usecases.}
\begin{tabular}{ |m{1.2cm} |m{1cm} |m{1.5cm} |m{1cm} |m{1.5cm} | }
 \hline
 & \multicolumn{2}{|c|}{SDR} & \multicolumn{2}{|c|}{SDN}\\
 \hline
 Usecase \# & number of processes & CPU utilization range & number of processes & CPU utilization range\\
 \hline\hline
 Usecase 1 & 50 & [80\%, 100\%] & 30 & [10\%, 30\%]\\
 \hline
 Usecase 2 & 30 & [60\%, 80\%] & 50 & [30\%, 50\%]\\
 \hline
 Usecase 3 & 50 & [80\%, 100\%] & 50 & [30\%, 50\%]\\
 \hline
\end{tabular}
\end{center}
\end{table}

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/CPU_cores_comparison.png}
\vspace{-0.1in}
	\caption{Number of CPU cores comparison}
	\label{fig:CPU_cores_comparison}
\end{figure}

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/power_consumption_comparison_80.png}
	\caption{Power consumption comparison for 80 CPU core server}

	\label{fig:power_consumption_comparison_80}
\end{figure}

\begin{figure}[h]
	\centering
\includegraphics[width=0.45\textwidth]{figures/power_consumption_comparison_400.png}
	\caption{Power consumption comparison for 400 CPU core server}
	\label{fig:power_consumption_comparison_400}
	\vspace{-0.2in}
\end{figure}

We compare the results of our algorithm with a baseline where \ac{SDR} and \ac{SDN} processes are allocated separately in different CPU cores.
Fig.~\ref{fig:CPU_cores_comparison} shows the boxplot of total number of CPU cores used. Twenty independent tests are randomly generated for each usecase with the parameter settings in Table~\ref{tab:usecases}.
From the results we can see that our scheme of \ac{SDN} and \ac{SDR} sharing the same group of CPU cores saves around 10\% - 20\% number of CPU cores, compared to the case of \ac{SDN} and \ac{SDR} utilizing CPU cores separately. The second use case has the most power saving percentage since in this use case the number of \ac{SDN} processes dominates, and they are easier to be parallelized, providing more room for grouping and consolidating processes in the cloud.

Fig.~\ref{fig:power_consumption_comparison_80} shows the total power consumption of these CPU cores. For this we have used power measurement data for the server model ASUSTeK Computer Inc. RS720-E10-R12, which contains 80 CPU cores, available at \cite{spec_power_asus}. Fig.~\ref{fig:CPU_cores_comparison} and Fig.~\ref{fig:power_consumption_comparison_80} shows similar trends, since the power vs CPU utilization relationship is close to linear \cite{spec_power_asus}. The power saving is around 10\% - 20\% for this server.

We also use a second power measurement dataset for a much larger server Hewlett Packard Enterprise Synergy 480 Gen10 Plus Compute Module, which contains 400 CPU cores, available at \cite{spec_power_hp}. The results are shown in Fig.~\ref{fig:power_consumption_comparison_400}. The trends are similar to the case of 80 CPU core system, but the power saving percentage is lower, which is around 5\% - 10\% percent. This is because the idle power consumption (i.e., the overhead) of this 400-core system is already 700 watt, which is much larger than the 100 watt idle power consumption (i.e., the overhead) of the 80-core system. The results in Fig.~\ref{fig:power_consumption_comparison_80} and Fig.~\ref{fig:power_consumption_comparison_400} are also boxplots generated from 20 independent tests.

There are other types of servers and CPUs that can be investigated to obtain power savings. In summary, our power saving scheme based on integrated \ac{SDN} and \ac{SDR} processing has up to 20\% power savings compared to the case where \ac{SDN} and \ac{SDR} are deployed in separate CPU cores.

\section{System Model}
\label{sec:system_model}

We propose an \ac{MEC} as an example for a use case of \ac{SDN} and \ac{SDR} integrated architecture, as shown in Figure~\ref{fig:MEC_architecture}. This architecture can be for example adopted in systems for inter-vehicle communications \cite{1543745}.
The architecture is all software-defined and network functions are virtualized. As the two major network functions for this architecture, \ac{SDN} and \ac{SDR} functions are integrated in the edge servers (i.e. \ac{MEC} nodes). Vehicular networks are illustrated at end user side. Inside the \ac{MEC} nodes, SDN switches, eNodeB \acp{BBU}, 4G/5G core function, and computational resources for \ac{MEC} tasks utilize the same group of servers. In this paper, we focus on the integration of the communication part, i.e., \ac{SDN} and \ac{SDR} components.

\begin{figure}[h]
	\centering
	\includegraphics[width=0.45\textwidth]{figures/overall_architecture.png}
	\caption{proposed MEC architecture}
	\label{fig:MEC_architecture}
	\vspace{-0.1in}
\end{figure}