\section{Introduction}
\label{sec:intro}

The scale of modern cellular networks and user service demands make performance optimisation more challenging than ever.
Network configuration must be adjusted automatically and in (near) real time to ensure a high level of Quality of Service (QoS) to each User Equipment (UE).
The Remote Electrical Tilt (RET) angle of antennas, defined as the change in the vertical orientation of antennas by an electrical design, is one of the most important variables to control for Self-Organising Networks (SONs). The goal is to find an optimal trade-off for some of the network Key Performance Indicators (KPIs), such as \textit{quality}, \textit{coverage} and \textit{capacity} \cite{Buenostado17}.
For example, an increase in antenna downtilt correlates with a stronger signal in a more concentrated area as well as higher capacity and reduced interference radiation towards other cells in the network.
However, excessive downtilting can result in insufficient coverage in a given area, with some UE unable to receive a minimum Reference Signal Received Power (RSRP).

Many existing solutions to downtilt adjustment use rule-based algorithms to optimise the tilt angle based on the historical network performance.
These rules are usually created by domain experts, and thus lack the scalability and adaptability required for modern cellular networks.
Developments in Reinforcement Learning (RL) \cite{kaelbling1996reinforcement} introduce novel data-driven solutions where RET optimisation is performed by an agent that continuously performs tilt angle adjustments and observes the changes in the environment in order to learn the optimal action for each network state \cite{razavi2010fuzzy, fan2014self, guo2013spectral, balevi2019online}.

However, when interacting with a sensitive environment, such as a cellular network, even small unsafe modifications can lead to drastic performance loss and disruption, which makes traditional RL solutions risky and impractical.
Therefore, RL has found limited application in mission-critical use cases.
These limitations lay grounds for the study of Safe Reinforcement Learning (SRL), where the RL agent is restricted in its exploration of the environment.
The restriction can be achieved by various safety mechanisms that minimise the adverse effects of unsafe actions \cite{garcia2015comprehensive}.
Our approach relies on existing baseline strategies that are known to be \textit{safe} with respect to a minimum level of performance, such as rules created by a domain expert.
When an action that is proposed by the RL agent performs worse than the baseline, it is deemed \textit{unsafe}, and the baseline action may be executed instead.
Hence, we define \textit{safety} as performing at least as well as a baseline whose performance is known to be acceptable.

In this paper, we propose a comprehensive modular architecture for SRL and demonstrate its viability on the RET optimisation use case.
The architecture consists of a safety shield, safety logic modules, a set of known safe baselines, and one or more RL agents.
The \textit{safety shield} wraps around the target environment with the goal of decoupling the RL agent from the environment, thereby preventing the RL agent from directly and uncontrollably affecting the environment.
The \textit{shield logic modules} provide methods to evaluate actions suggested by RL agents and baselines, and to choose which action should be performed on the environment.
The \textit{baselines} are considered safe on at least a subset of the state-action space (although with potentially sub-optimal performance).
For example, an individual baseline may be limited to a particular area of the network, and therefore
is only considered safe for that area, while other baselines may be safe for other areas.
Similarly, different \textit{RL agents} may follow different models or have different hyper-parameters.
In those circumstances, our approach benefits from the ability to compare multiple agents and baselines.
The modularity of the proposed solution enables customisation by modifying the shield logic module based on the use case and available RL agents and baselines.

\section{Related Work}
\label{sec:related_work}

Recently, many works have explored the use of RL methods as a solution for the RET optimisation problem \cite{guo2013spectral,balevi2019online,razavi2010fuzzy,fan2014self}.
In \cite{guo2013spectral}, the authors maximise the UE's throughput fairness and energy efficiency of the network using an RL-based approach.
\cite{balevi2019online} addresses a similar problem while expanding the scope to a group of cells, each with its own antenna configuration, resulting in an extremely complex problem.
Both \cite{razavi2010fuzzy} and \cite{fan2014self} use Fuzzy Reinforcement Learning methods, utilising fuzzy control theory through the use of predefined rules in order to model the inherent uncertainty of the observed states of cellular networks and reduce the state dimensionality.
The main limitation with the majority of this work is that the solutions propose potentially unsafe actions that can have an adverse impact on the network.

Therefore, several methods have been proposed to achieve safety in the context of Safe Reinforcement Learning (SRL) \cite{garcia2015comprehensive}.
Examples of such safe strategies include the use of accumulated past knowledge \cite{ghavamzadeh2016safe}, making conservative action choices that prioritise avoiding the worst-case scenario, or requesting guidance from an external agent or a human operator when the current state is considered too risky \cite{torrey2012help, reddy2019learning}.
In \cite{alshiekh2018safe}, the definition of shielding is introduced where temporal logic, a model-based method, is proposed to filter out unsafe actions proposed by the RL agent.
In \cite{vairamuthu2020antenna}, the authors consider an SRL method based on an actor-critic model where a safety threshold $\varepsilon$ is progressively adjusted during the training, which only allows actions that are close to that of a safe baseline policy. The authors of \cite{Vannella21} tackle RET optimisation using an offline RL algorithm that modifies the exploration based on the information contained in a batch of data. Such method can be framed as a particular instance of our architecture with a single baseline policy and a single learning agent.

In \cite{zhang2019towards}, the authors use a linear combination of suggested actions from the RL agent or the baseline with a parallel pioneer policy to prevent drastic changes in the control policy.
Many of these works are tied to a specific RL algorithm, and none of them consider multiple baselines for increased safety in state spaces where an individual baseline might be inadequate.
Our approach tackles these shortcomings in the current state of the art by proposing an SRL architecture that provides the flexibility to plug in various types of RL agents, baselines and decision logics towards better safety guarantees.

\section{Reinforcement Learning}
\label{sec:background}

Reinforcement Learning (RL) \cite{kaelbling1996reinforcement} is a long-established decision-making framework, in which an agent interacts with an environment by exploring its states and selecting actions to maximise the long-term return based on a reward signal.
An RL agent follows a policy $\pi$ choosing the action to be performed at each state.
The agent devises an optimal policy by dynamically optimising a value function that is estimated based on previous interactions of the RL agent.
At each time step, the agent can choose to either follow the recommended action from the estimated best action (exploitation), or choose a random action to gather further information about the environment and update its knowledge (exploration).

There are multiple approaches to find the optimal policy.
$Q$-learning is one of the most popular RL algorithms that finds the optimal policy by estimating the state-action value function ($Q$-function) $Q^{\pi(s,a)}$, representing the expected cumulative reward when being in state $s$, executing action $a$, and following policy $\pi$ afterwards.
When Artificial Neural Networks (ANNs) are used to parameterise the $Q$-function, the algorithm is known as Deep $Q$-Network (DQN) \cite{mnih2015human}. DQN uses optimisation methods such as Stochastic Gradient Descent (SGD) to minimise Mean Squared Error (MSE) between target values
and the parameterised $Q$-function.
Another popular approach to find the optimal policy is the Actor-Critic (AC) algorithm. AC  uses ANN to parameterise two networks: the \textit{actor network}, taking as input the state and returning the action to be performed on the environment, and the \textit{critic network}, evaluating the action proposed by the actor by computing the value function. In this study we investigate DQN and AC agents for our proposed SRL architecture.

\section{A Safety Shield Architecture for Safe Reinforcement Learning}
\label{sec:architecture}

Our approach is an SRL model that uses a modular shield architecture with one or more RL agents and baselines.
Figure \ref{fig:architecture} illustrates an overview of the proposed modular SRL structure.
It comprises a set of RL agents that are benchmarked against safe baselines through a safety shield that chooses the action to be performed on the environment.
The architecture components are the following (numbers refer to Figure \ref{fig:architecture}):

\begin{enumerate}[(1)]
    \item \textbf{Environment}: This component represents the environment that the agent is interacting with, modelled as a standard RL problem \cite{kaelbling1996reinforcement}.
    \item \textbf{Safety Shield}: This component is the mediator between the RL agents (3) and the environment (1).
    Contrary to the traditional RL model, where agents directly interact with the environment and receive a feedback, the safety shield acts as a proxy between agents and the environment to protect it from possibly unsafe actions proposed by the agents.
    The safety shield collects proposed actions by agents and baselines (8) and chooses a final safe action (6) to be performed on the real environment.
    The environment feedback (7) includes the action that is actually performed by the shield on the environment and is fed back to agents and baselines through the safety feedback (10).

    \item \textbf{RL Agents}: A set of RL agents indirectly interact with the environment by proposing actions (8) to the safety shield.
    The agents are continuously trained during the RL interaction to improve their future recommended actions.
    Using different learning methods enables parallel experimentation with multiple techniques and parameters, to determine the suitable safe action amongst them.

    \item \textbf{Safe Baselines}: Baselines also receive feedback about the current state of the environment to suggest future actions.
    Contrary to the RL agent actions, baseline actions are considered safe at least on a subset of the state-action space.
    Examples of safe baselines are rule-based algorithms that recommend actions satisfying safety rules designed by domain experts and supervised models trained on previous safe actions.

    The baselines are considered sub-optimal in terms of maximum performance but safe with regards to the minimum level of performance they provide at any given time.
    \item \textbf{Safety Logic}: The logic provides the safety constraints (9) to the safety shield, which are used to select the safe action from agents and baselines.
    The architecture supports various safety logic implementations, for example $(i)$ using only the proposed actions from agents and baselines, where the safety logic chooses the best action by some safety criteria,

    or $(ii)$ using agent and baseline internal variables such as convergence as additional information to gauge the safety of the proposed action.

    This is enabled through the feedback function. The shield logics investigated in this study are of type $(i)$.

\end{enumerate}

\begin{figure}[tbp]
\centering
    \vspace{0.04in}
    \includegraphics[width=.5\textwidth]{figures/architecture.pdf}
\caption{SRL architecture with safety shield, logic and multiple baselines.}
\label{fig:architecture}
\end{figure}

\section{Antenna Tilt Optimisation using SRL}
\label{sec:safe-rl-ret}

{The definition of RET optimisation problem follows previous works for \textit{Coverage-Capacity Optimization} (CCO) \cite{Buenostado17, Vannella20}, where the goal is to maximise network \textit{coverage} and \textit{capacity}, while minimising inter-cell \textit{interference}. We define network measurements for capacity and coverage KPIs for time $t$ and cell $c$ to be $\textsc{Cov}_{t,c}$ and $\textsc{Cap}_{t,c}$ respectively. Also, the effect of negative interference on neighbouring cells is modelled by the {quality} KPI $\textsc{Qual}_{t,c}$. Note that above metrics are considered risk KPIs, i.e. high values indicate high risk for insufficient coverage, capacity or quality in a given cell.}

\begin{table}[tbp]
    \centering
    \vspace{0.04in}
    \caption{Parameters for the simulator and the RL training.}
    \label{tab:parameters}
    \begin{tabular}{lr}
        \hline
        \textsc{Simulator parameter} & \textsc{Value} \\

        \hline

        Number of base stations & $7$ \\

        Number of cells & $21$ \\

        Number of user equipments   & $2000$ \\

        Frequency  & $2$ GHz \\

        Traffic volume & $20$ Mbps \\

        Antenna height & $32$ m \\

        Minimum downtilt angle & $1^{\circ}$\\

        Maximum downtilt angle & $16^{\circ}$\\%[5pt]

        \hline
        \textsc{DQN Hyper-parameter}       & \textsc{Value} \\

        \hline

        Discount factor & $0.0$ \\

        Learning rate & $0.001$ \\

        Batch size & $50$ \\

        Test episode length & $20$ \\

        Number of evaluation episodes & $25$ \\%[5pt]

        \hline
        \textsc{AC Hyper-parameter}       & \textsc{Value} \\
        \hline

        Discount factor & $0.0$ \\

        Learning rate & $0.03$ \\

        Test episode length & $20$ \\

        Number of evaluation episodes & $25$ \\%[5pt]
        \hline
    \end{tabular}
\end{table}

We now formulate the RET optimisation problem as a use case for the SRL architecture introduced in the previous section, with the following components:

\begin{itemize}
    \item \textit{Environment}: consists of a simulated mobile network environment in an urban area. The parameters of this environment are detailed in Table \ref{tab:parameters}.
    Further, to avoid local optima, the cell downtilt values are reset to random values at the end of each episode.

  \item \textit{State space:} $\mathcal{S} \subseteq [0,1]^4$: comprises normalised values of \textit{downtilt} $\theta$, \textit{coverage} \textsc{Cov}, \textit{capacity} \textsc{Cap} and \textit{quality} \textsc{Qual} for time $t$ and cell $c$., i.e. $s_{t,c} = [\theta_{t,c},\textsc{Cov}_{t,c},\textsc{Cap}_{t,c}, \textsc{Qual}_{t,c}]$.
  \item \textit{Action space:} $\mathcal{A} =\{-1,0,1\}$: consists of discrete changes of the current downtilt. At time $t$ and at cell $c$ the agent selects $a_{t,c}\in\mathcal{A}$.
  \item \textit{Reward:} $r_{t,c} = -\log(1 + \textsc{Cov}_{t,c}^2 +\textsc{Cap}_{t,c}^2 + \textsc{Qual}_{t,c}^2)$: consists of the squared log-sum of coverage, capacity, and quality risk KPIs.
  \item \textit{Safe baselines}: we considered two baselines for the experiments conducted for this study:
    \begin{itemize}
        \item \textit{Rule-based baseline:} this baseline is based on a legacy RET algorithm currently in place for antenna tilt optimisation that suggests antenna tilt actions based on $\textsc{Cov}_{t,c}$, $\textsc{Cap}_{t,c}$ and $\textsc{Qual}_{t,c}$.
        \item \textit{Model-based baseline:} this baseline is generated from a DQN-based offline learner trained on historical data of performed antenna tilts.
     \end{itemize}
  \item \textit{RL Agents:} In the experiments conducted for this study, we use an RL agent using the DQN or the AC algorithm.
  The training parameters of these agents
  are
  given in Table \ref{tab:parameters}. {The same policy is independently applied to all cells to suggest a specific action for each cell.}
\end{itemize}

In this setting, we study the performance of two shield logics, namely State Predictor Shield Logic and $k$-Shield Logic, which are described below.

\subsection{State Predictor Shield Logic}
\label{sec:state-predictor}

The State Predictor Shield Logic uses a pre-trained supervised learning model to assess the value of the actions proposed by the different RL agents and baselines.
    We train a multi-target regressor on historical data from the environment, where the input for the model is defined as a given environment state plus a tilt change action $(\textsc{Cov}_{t,c}, \textsc{Cap}_{t,c}, \textsc{Qual}_{t,c}, a_{t,c})$, and the output is defined as the environment state after applying the action $(\textsc{Cov}_{t+1,c}, \textsc{Cap}_{t+1,c}, \textsc{Qual}_{t+1,c})$.
The regressor in our experiments is a fully connected multi-layer perceptron
We also experimented with extended state-space, including additional KPIs, but that did not noticeably impact the prediction accuracy of the model.

With a sufficiently large sample size, a supervised model can be trained that captures the impact of various antenna angles on the measured network KPIs with reasonable accuracy.
This model is of course subject to the limitations of supervised learning that RL is trying to solve, such as decreased accuracy on unseen data, because supervised learning is susceptible to wrong/incomplete data and cannot adapt from interaction with the environment.
For the results reported in this work, we train the supervised State Predictor model on data that we synthesised from the simulation environment by randomising environment parameters to create a varied set of data points with different antenna tilt angles and states.
This way, the data for the supervised model follows a similar distribution to the data the RL is operating on.
However, we also experimented with a supervised model trained on historical data from a real network deployment, and we found that it performed on a similar level compared to the one trained on synthetic data.
This underlines the viability of using supervised learning models, even from a different deployment environment, for the purpose of increasing safety of the RL approach.

The State Predictor Shield Logic uses this pre-trained model to compare the actions proposed by agents and baselines by querying the model with each action in addition to the current environment state.
From the predicted next state, the shield logic can calculate various metrics to score the proposed actions, such as a reward or a safety constraint.
In our experimental setup, the safety logic compares proposed actions by using the state KPI directly.
It will choose the action that achieves the best predicted state KPI and forwards it to the shield instance for execution on the environment.

\subsection{$k$-Shield Logic}
\label{sec:K-Shield}
The logic in this scenario is inspired by the approach proposed in \cite{zhang2019towards} and is generalised to accommodate multiple
safe baselines.
It chooses an action by selecting either the RL agent's action or one of the baselines' action with a given probability, gradually switching control over from the baselines to the RL agent.
Formally, the control policy $\pi_\mathcal{C}$ is chosen between the RL policy $\pi_L$ and a set of $B$ safe baseline policies $\{\pi_s^1, \dots, \pi_s^B\}$ as follows:

\begin{equation}

    \pi_\mathcal{C} = p \pi_s^i + (1-p) \pi_L.
\end{equation}

When selecting between the safe baselines or the RL policy, we sample $p\sim \mathcal{B}(k)$, i.e. a Bernoulli random variable with parameter $k \in [0,1]$.
The parameter $k$ determines the probability of choosing (one of) the safe baselines over the RL policy.
The probability $k$ is initially close to $1$ to ensure that the control policy is dominated by safe baselines when the RL policy has not accumulated enough experience to be able to execute safe actions.
Subsequently, its value is reduced by a diminishing factor $d\in (0,1)$ every time the reward during previous episodes is improved compared to the past $w$ episodes.
Each episode is indexed as $e = 1,\dots, E$. The update is executed each $w$ episodes as:
\begin{equation}
  k_{e+1} =
    \begin{cases}
      \max\{0,k_e - d\} & \hat{R}_{[e-w,e]} \geq \hat{R}_{[e-2w,e-w-1]} \\
      k_e & \text{otherwise}
    \end{cases}
\end{equation}
where $\hat{R}_{[l,m]}$, denotes the average reward calculated from episode $l$ to  episode $m$.

The categorical random variable $i \sim \textsc{Cat}(b)$, with parameter $b \in [0,1]^B: \sum_{i=1}^B b_i = 1$, controls the importance of each baseline. If $p=1$, we select the $i$-th baseline with probability $b_i$.
The baselines probabilities $b_i$ can be:
\begin{itemize}
    \item preset values based on the performance of safe baselines, and defined by domain experts or use-case stakeholders (e.g., network operators in RET optimisation), or
    \item state-action dependent, allowing higher weights for policies that outperform others in certain sub-spaces of the problem (e.g., if a safe baseline has higher performance on a sub-region of a cellular network).
\end{itemize}

In the experiments conducted for this study, we investigate scenarios where an RL agent competes with a single baseline and with multiple baselines for different values of $b$.
The results of our experimentation
are detailed in the next section.

\section{Evaluation Results}
\label{sec:experiment_results}

The results discussed in this section present the outcomes of our shielded SRL experiments using the modular shield architecture described in Section \ref{sec:architecture}.
We run each experiment with $6$ random seeds, and
each plot shows the average of all random seeds (continuous and dashed lines), as well as the minimum and maximum values that were reached over all seeds (shaded areas).
We apply a running average window to smooth out the curves for better visual comprehension.
Finally, we focus on the early phase of RL training, in which exploration is prevalent and therefore most unsafe behaviour of an unrestricted RL agent is to be expected.
The long-term performance advantages of RL over traditional approaches have already been shown in the literature, e.g. \cite{guo2013spectral,razavi2010fuzzy,fan2014self}.

The following elements will be shown in the plots:
\begin{enumerate}
    \item Unrestricted RL agents with full environment access,
    \item Baselines that are considered sub-optimal but safe,
    \item SRL agents that are governed by the safety shield using a shield logic.
\end{enumerate}
The plots focus on reward (higher is better), and coverage and quality risk KPIs (lower is better).
The capacity KPI is omitted as it was not impacted by antenna tilts across all experiments.

\subsection{Comparison of unrestricted RL agent with baseline and SRL agent}
In this section we compare the performance of unrestricted RL agents with baselines and with the SRL agents using the two shield logics described in sections \ref{sec:state-predictor} and \ref{sec:K-Shield}.

The State Predictor Shield Logic uses a pre-trained machine learning model to estimate the impact of the proposed tilt change action on the network as described in Section \ref{sec:state-predictor}.
Figure \ref{fig:state_predictor_results} (top) shows the immediate reward that is returned by the environment.
We observe that the unrestricted DQN agent (solid blue) starts off with a sub-optimal reward, which is significantly worse than the performance of the safe baseline.
When the safety shield is applied, the safe DQN agent (dashed blue) closely mimics the baseline behaviour and slightly outperforms the baseline early on.
This is the desired effect that the safety shield with the State Predictor Shield Logic was designed to accomplish.

Figure \ref{fig:state_predictor_results} (middle and bottom) focuses on coverage and quality, which are two important performance indicators.
In our experiments, coverage and quality are risk-based KPIs, which means that lower values indicate better network conditions.
In both cases, it is apparent that the unrestricted DQN agent starts off at a much higher risk level before it learns the environment and its performance is improved, although the effect is more pronounced on the coverage KPI.
The safe baseline (orange) is at a safe level from the start with little fluctuation, where the safe level of performance is set by network operator experts.
The State Predictor Shield Logic causes the DQN agent to follow the baseline performance more closely from the start, thus creating a safe agent interaction with the environment.
In the case of coverage, the safe agent slightly outperforms the baseline on average early on.

Before comparing the performance of the $k$-Shield Logic, we first tune the diminishing factor $d$ and the average reward window size $w$ as described in Section \ref{sec:K-Shield}. We compared the performance of the safe DQN agent using $k$-Shield Logic for $d \in \{ 0.1, 0.2 \}$ and $w \in \{2,5\}$. While the results show very small difference,
a smaller $d$ indicates a more gradual transfer to RL agent's policy, allowing to rely on safe baselines for a longer period of the training.
Also, smaller $w$ indicates more frequent comparison between average rewards over previous episodes, which allows quicker adaptation to current rewards, which in turn improves the performance of the agent governed by the $k$-Shield Logic.

\begin{figure}[tbp]
    \centering
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/statepredictor_reward.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_reward.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/statepredictor_coverage.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_comparison_coverage.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/statepredictor_quality.pdf}
        \caption{State Predictor Shield Logic.}
        \label{fig:state_predictor_results}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_comparison_quality.pdf}
        \caption{$k$-Shield Logic.}
        \label{fig:k_shield_results}
    \end{subfigure}
    \caption{Comparison of reward (higher is better), coverage and quality risk KPIs (lower is better) for DQN and AC using unrestricted agents, baseline only and safe agents governed by the shield logic. Shaded areas represent min/max over all runs.}
    \label{fig:comparison_results}
\end{figure}

Following this observation, $d=0.1$ and $w=2$ are selected as optimal parameter values that will be used for the $k$-Shield Logic experiments. Figure \ref{fig:k_shield_results} (top) shows the reward that is returned by the environment over time. As it can be observed, the safe DQN agent with $k$-Shield Logic (dashed blue) follows the safe baseline policy during the period in which the performance of unrestricted DQN (blue) is inferior to that of the the safe baseline. It then gradually switches towards unrestricted DQN agent as the model is further trained to achieve greater rewards than the safe baseline.
Similar to the State Predictor Shield Logic, for the coverage and quality risk KPIs (Figure \ref{fig:k_shield_results} middle and bottom), the $k$-Shield Logic prevents the DQN agent from performing high-risk actions by following the safe baseline policy at the start of the training phase, where the DQN agent performs mostly exploratory actions, and subsequently switching to the DQN actions when it has gathered enough experience to execute a safe action. This behaviour is more visible for coverage.

We also compare the unrestricted and safe AC agent to the unrestricted and safe DQN agent for both the State Predictor Shield Logic (Figure \ref{fig:state_predictor_results}, green) and the $k$-Shield Logic (Figure \ref{fig:k_shield_results}, green).
We observe that the trend for reward, coverage and quality KPIs when using a DQN agent is similar to when the safety logic is used to restrict actions of the AC agent, although the range of values for the two agent types differs.
Both agent types converge during the experiments in terms of loss, although the shield tends to delay the convergence for the SRL agents slightly.
We omitted the loss plots for brevity.

\subsection{Comparison of multiple baselines}

\begin{figure}[tbp]
    \centering
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/two_baselines_statepredictor_reward.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_multiple_baselines_reward.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/two_baselines_statepredictor_coverage.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_multiple_baselines_coverage.pdf}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/two_baselines_statepredictor_quality.pdf}
        \caption{State Predictor Shield Logic.}
        \label{fig:two_baselines_state_predictor_results}
    \end{subfigure}
    \begin{subfigure}[t]{0.49\columnwidth}
        \includegraphics[width=\textwidth]{figures/kshield_multiple_baselines_quality.pdf}
        \caption{$k$-Shield Logic.}
        \label{fig:two_baselines_kshield_results}
    \end{subfigure}
\caption{Comparison of reward (higher is better), coverage and quality risk KPIs (lower is better) for shield-logic governed DQN using rule-based, model-based and both baselines. Shaded areas represent min/max over all runs.}
\label{fig:two_baselines_results}
\end{figure}

In this section, we compare different baselines in a scenario where the shield logic has access to multiple baselines and can choose between them according to its strategy.
The two baselines are Rule-based and Model-based, as described in Section \ref{sec:safe-rl-ret}.
To keep the plots readable, these experiments focus on the DQN agent only.

Figure \ref{fig:two_baselines_state_predictor_results} shows a comparison of the received rewards and the achieved risk KPIs when training a DQN learner on a shielded environment where the State Predictor Shield Logic chooses between different baselines.
In the network environment considered for this study, the model-based baseline (dashed orange) performs worse than the alternative baseline (dashed blue).
However, when providing the State Predictor Shield Logic with both baselines, it is able to select the best proposed action based on the predicted state (green).

Figure \ref{fig:two_baselines_kshield_results} compares the reward, and coverage and quality risk KPIs of a $k$-Shield Logic restricted DQN agent using both Rule-based and Model-based baselines for different importance weights.
It can be seen that in this scenario, where the Rule-based baseline is outperforming the Model-based baseline, higher weights on the Rule-based baseline result in faster convergence to better reward (top) and coverage (middle). This is less apparent for quality (bottom) as the difference between the performance for different scenarios is very small and within the margin of error.

\section{Conclusions}
\label{sec:conclusions}

In this paper, we propose a modular architecture for Safe Reinforcement Learning (SRL) that enables safe interaction with the environment by benchmarking recommendations of RL agents to multiple safe baselines.
The architecture allows experimentation with multiple RL agents, using different models and hyper-parameters, and leverages multiple baselines that may be safe in subsets of the state-action space, to create a policy that is safe across the whole state-action space and outperforms individual unrestricted RL agents and baselines.
As a concrete use case for our architecture, we study RET optimisation, which is evaluated using a simulated environment that is configured to recreate a real-world urban network.

The safety and performance of the approach using multiple shield logics is compared with an existing rule-based SON algorithm and an offline model that is trained on historical data.
The results show that, compared to safe baselines, unrestricted RL agents explore unsafe regions of the state-action space, especially at the start of the RL learning process.
The safety shield allows RL agents to start learning in safe states (performing similar to safe baselines) with the help of the shield logics, and eventually outperform safe baselines in most cases, especially for the received reward and the coverage KPI.
A limitation to this approach is the availability of safe baselines, without which the safety shield would not be able to decide whether a proposed action is safe enough for the environment.

\bibliographystyle{IEEEtran}


\section{Introduction}

Recently, automatically generating image descriptions has attracted considerable interest in the fields of computer vision and nature language processing. Such a task is easy to humans but highly non-trivial for machines as it requires not only capturing the semantic information from images (e.g., objects and actions) but also needs to generate human-like natural language descriptions.

Existing approaches to generating image description are dominated by neural network-based methods, which mostly focus on generating description for a single image \cite{Karpathy2015Deep,Xu2015Show,Jia2015Guiding,You2016Image}. Generating descriptions for sequential images, in contrast, is much more challenging, i.e., the information of both  individual images as well as the dependencies between images in a sequence needs to be captured.

\citet{Huang2016} introduce the first sequential vision-to-language dataset and exploit  Gated Recurrent Units (GRUs) \cite{Cho2014Learning} based encoder and decoder for the task of visual storytelling. However, their approach only considers image information of a sequence at the first time step of the decoder, where the local attention mechanism is ignored which is important for capturing the correlation between the features of an individual image and the corresponding words in a description sentence. \citet{Yulicheng2017} propose a hierarchically-attentive Recurrent Neural Nets (RNNs) for album summarisation and storytelling. To generate descriptions for an image album, their hierarchical framework selects representative images from several image sequences of the  album, where the selected images might not necessary have correlation to each other.

\begin{figure*}
\centering
\includegraphics[width = 0.8\textwidth]{fram.pdf}
\caption{The architecture of our CNN-LSTM model with global semantic context.} \label{fig:fram}
\end{figure*}

In this paper, we propose an end-to-end CNN-LSTM model with a local-object attention mechanism for generating story-like descriptions for multiple images of a sequence. To improve the coherence of the generated descriptions, we exploit a  paralleled long short-terms memory (LSTM) network and learns global semantic context by embedding the global features of sequential images as an initial input to the hidden layer of the LSTM model.
We evaluate the performance of our model on the task of generating story-like descriptions for an image sequence on the sequence-in-sequence (SIS) dataset published by Microsoft. We hypothesise that by taking into account global context, our model can also generate better descriptions for individual images. Therefore, in another set of experiments, we further test our model on the  Descriptions of Images-in-Isolation (DII) dataset for generating descriptions for each individual image of a sequence.
Experimental results show that our model outperforms a baseline developed based on the state-of-the-art image captioning model~\cite{Xu2015Show} in terms of BLEU, METEOR and ROUGE, and can generate sequential descriptions which preserve the dependencies between sentences.

\section{Related Work}

Recent successes in machine translation using Recurrent Neural Network (RNN) \cite{Bahdanau2014Neural,Cho2014Learning} catalyse the adoption of neural networks in the task of image caption generation. Early works of image caption generation based on CNN-RNN networks have been made great progress.\citet{Vinyals2014Show} propose an encoder-decoder model which utilises a Convolutional Neural Network (CNN) for encoding the input image into a vector representation and a Recurrent Neural Network (RNN) for decoding the corresponding text description. Similarly, \citet{Karpathy2015Deep} present an alignment model based on a CNN and a bidirectional RNN which can align segment regions of an image to the corresponding words of a text description.
\citet{Donahue2014} propose a Long-term Recurrent Convolutional Network (LRCN) which integrates convolutional layers and long-range temporal recursion for generating image descriptions.

Recently, the attention mechanism \cite{Xu2015Show,You2016Image,Lu16,Zhou2016} has been widely used and proved to be effective in the task of image description generation. For instance, \citet{Xu2015Show} explore two kinds of attention mechanism for generating image descriptions, i.e., soft-attention and hard-attention, whereas \citet{You2016Image} exploits a selective semantic attention mechanism for the same task.

There is also a surge of research interest in visual storytelling \cite{Kim2014,Sigurdsson2016,Huang2016,Yulicheng2017}. \citet{Huang2016} collect stories using Mechanical Turk and translate a sequence of images into story-like descriptions by extending a GRU-GRU framework. \citet{Yulicheng2017} utilise a hierarchically-attentive structures with combined RNNs for photo selection and story generation. However, the above mentioned approaches for generating descriptions of sequential images do not explicitly capture the dependencies between each individual images of a sequence, which is the gap that we try to address in this paper.

\section{Methodology}

In this section, we describe the proposed CNN-LSTM model with local-object attention. In order to generate coherent descriptions for an image sequence, we introduce global semantic context and a paralleled LSTM in our framework as  shown in Figure.~\ref{fig:fram}. Our model works by first extracting  the global features of sequential images using a CNN network (VGG16) \cite{Simonyan2014Very}, which has been extensively used in image recognition. Here a VGG16 model contains 13 convolutional layers, 5 pooling layers and 3 fully connected layers. The extracted global features are then embedded into a global semantic vector with a multi-layer perceptron as the initial input to the hidden layer of a paralleled LSTM model. Our model then applies the last convolutional-layer operation from the VGG16 model to generate the local features of each image in sequence. Finally, we introduce a paralleled  LSTM model and a local-object attention mechanism to decode sentence descriptions.

\subsection{Features Extraction and Embedding}
Sequential image descriptions are different from single image description due to the spatial correlation between images. Therefore, in the encoder, we exploit both global and local features for describing the content of sequential images. We extract global features of the sequential images with the second fully connected layer (FC7) from VGG16 model. The global features are denoted by $G$ which are a set of 4096-dimension vectors. Then, we select the features of the final convolutional layer ($Cov\_5$) from the VGG16 model to represent local features for each image in the sequence. The local features are denoted as $L_j$ ($j=1$,$\dots$,$N$), where $N$ is the number of images in the sequence. In our experiment, we follow \citet{Huang2016} and set 5 as the number of images in a sequence. Finally, we embed the global features $G$ into a 512-dimension context vector via a multi-layer perceptron which is then used as the initial input of the hidden layer in LSTM model.

\subsection{Sequential Descriptions Generation}

In the decoding stage, our goal is to obtain the most likely text descriptions of a given sequence of images. This can be generated by training a model to maximize the log likelihood of a sequence of sentences $S$, given the corresponding sequential images $I$ and the model parameters $\theta$, as shown in Eq.~\ref{eq:probability}.

\begin{equation} \label{eq:probability}
\theta^* = \mathop{\arg\max}_{\theta} \sum_{j=1}^N \sum_{(I,s_j)} log\, p(s_j|I,\theta)
\end{equation}
Here $s_j$ denotes a sentence in $S$, and $N$ is the total number of sentences in $S$.

Assuming a generative model of each sentence $s_j$  produces each word in the sentence in order, the log probability of $s_j$ is given by the sum of the log probabilities over the words:
\begin{equation}
\log p(s_j|I) = \sum_{t=1}^C log\, p(s_{j,t}|I,s_{j,1},s_{j,2}...s_{j,t-1})
\end{equation}
where $s_{j,t}$ represents the $t^{th}$ word in the $j^{th}$ sentence and $C$ is the total number of words of $s_j$.

We utilize a LSTM network \cite{Hochreiter1997Long} to produce a sequence descriptions conditioned on the local feature vectors, the previous generated words, as well as the hidden state with a global semantic context. Formally, our LSTM model is formulated as follows:
\begin{eqnarray}
&& i^j_t = \sigma(W_{xi}x^j_{t-1}+W_{hi}h^j_{t-1}+W_{vi}v^j_t+b_i) \nonumber\\
&& f^j_t = \sigma(W_{xf}x^j_{t-1}+W_{hf}h^j_{t-1}+W_{vf}v^j_t+b_f) \nonumber\\
&& o^j_t = \sigma(W_{xo}x^j_{t-1}+W_{ho}h^j_{t-1}+W_{vo}v^j_t+b_o) \nonumber\\
&& q^j_t = \varphi(W_{xq}x^j_{t-1}+W_{hq}h^j_{t-1}+W_{vq}v^j_t+b_q) \nonumber\\
&& c^j_t = f^j_t \odot c^j_{t-1} + i^j_t \odot q^j_t \nonumber\\
&& h^j_t = o^j_t\odot \varphi(c^j_t) \label{eq3}
\end{eqnarray}
where $i^j_t$, $f^j_t$, $o^j_t$ and $c^j_t$ represents input gates, forget gates, output gates and memory, respectively. $q^j_t$ represents the updating information in the memory $c^j_t$. $\sigma$ denotes the sigmoid activation function, $\odot$ represents the element-wise multiplication, and $\varphi$ indicates the hyperbolic tangent function. $W_\bullet$ and $b_\bullet$ are the parameters to be estimated during training. Also $h^j_t$ is the hidden state at time step $t$ which will be used as an input to the LSTM unit at the next time step.

Here, we utilize a multilayer perceptron to model the global semantic context which can be viewed as the initial input of the hidden state $h^j_0$, where every initial value $h^j_0$ in the LSTM model is equal and is defined as:
\begin{equation}
 h^j_0 = W_0 \, \varphi(W_gG+b_g)
\end{equation}

\begin{figure*}
\centering {\includegraphics[width = 1.0\textwidth]{finalcaption.pdf}}
\caption{Example of sequential descriptions generated by our model, the baseline, and the ground truth.} \label{fig:finalcaption}
\end{figure*}

When modelling local context, the local context vector $v^j_t$ is a dynamic representation of the relevant part of the $j^{th}$ image in a sequence at time $t$. In Eq.~\ref{eq:attention}, we use the attention mechanism $ f_{att} $ proposed by \cite{Bahdanau2014Neural} to compute the local attention vector $v^j_t$, where the corresponding weight $k^j_t$ of each local features $L_j$ is computed by a softmax function with input from a multilayer perceptron which considers both the current local vector $L_j$ and the hidden state $h^j_{t-1}$ at time $t-1$.
\begin{equation}
 k^j_t = softmax(W_k\,tanh(W_{lv}L^j+W_{hv}h^j_{t-1}+b_v))
\end{equation}
\begin{equation} \label{eq:attention}
 v^j_t = \sum_{i=1}^M k^j_{it} L^j_i
\end{equation}

\begin{figure*}
\centering {\includegraphics[width = 1.0\textwidth]{sis_caption.pdf}}
\caption{Error analysis of our model. First row: our model generates correct captions. Second row: failure cases due to severe overfitting.} \label{fig:sis_caption}
\end{figure*}

\section{Experiments}

\noindent {\bf Dataset.}~~~

Both the SIS and DII datasets are published by Microsoft\footnote{\url{http://visionandlanguage.net/VIST/}}, which have a similar data structure, i.e., each image sequence consists of five images and their corresponding descriptions. The key difference is that descriptions of SIS consider the dependencies between images, whereas the descriptions of DII are generated for each individual image, i.e., no dependencies are considered. As the full DII and SIS datasets are quite large, we only used part of both datasets for our initial experiments, where the dataset statistics are shown in Table~\ref{tb:dataset}.

\begin{table}[t!]
\begin{center} \small
\begin{tabular}{|c|c|c|c|}
    \hline \bf Dataset & \bf Train & \bf Test   & \bf Vocab. Size  \\
    \hline DII  & 23,415 & 1,665 &  10,000  \\
	 \hline SIS  & 110,905  & 10,370  & 18,000    \\
     \hline
\end{tabular}
\end{center}
\caption{\label{tb:dataset} Dataset statistics.}
\end{table}

\noindent {\bf Evaluation.}~~~We compare our model with the sequence-to-sequence baseline (cnn-att-lstm) with attention mechanism \cite{Xu2015Show}.  The cnn-att-lstm baseline only utilises the local attention mechanism which combines visual concepts of an image with the corresponding words in a sentence. Our model, apart from adopting a local-object attention, can further model global semantic context for capturing the correlation between sequential images.
\begin{table}[t!]
\begin{center} \small
\begin{tabular}{|c|c|ccc|}
    \hline \bf Dataset & \bf Method & \bf BLEU & \bf METEOR & \bf ROUGE\\
    \hline
    \multirow{2}{*}{DII} & cnn-att-lstm &  36.1  & 9.2  & 26.9    \\
	     & Our model   & \bf 40.1 & \bf 11.2  & \bf 29.1 \\
     \hline  \multirow{2}{*}{SIS}& cnn-att-lstm &  15.2 & 4.6  & 13.6  \\
       & Our model  & \bf 17.2  &  \bf 5.5 & \bf 15.2  \\
	\hline
\end{tabular}
\end{center}
\caption{\label{tb:metrics}Evaluation of the quality of descriptions generated for sequential images.}
\end{table}

Table~\ref{tb:metrics} shows the experimental results of our model on the task of generating descriptions for sequential images  with three popular evaluation metrics, i.e. BLEU, Meteor and ROUGE. It can be observed from Table~\ref{tb:metrics} that our model outperforms the baseline on both SIS and DII datasets for all evaluation metrics. It is also observed that the scores of the evaluation metric are generally higher for the DII dataset than the SIS dataset. The main reason is that the SIS dataset contains more sentences descriptions in a sequence and more abstract content descriptions such as ``breathtaking'' and ``excited'' which are difficult to understand and prone to overfitting.

Figure~\ref{fig:finalcaption} shows an example sequence of five images as well as their corresponding descriptions generated by our model, the baseline (cnn-att-lstm), and the ground truth.
For the SIS dataset, it can observed that our model can capture more coherent story-like descriptions. For instance, our model can learn the social word ``family'' to connect the whole story and learn the emotional words ``great time'' to summarise the description. However, the baseline model failed to capture such important information.
Our model can learn dependencies of visual scenes between images even on the DII dataset. For example, compared to the descriptions generated by cnn-att-lstm, our model can learn the visual word ``beach'' in image 1 by reasoning from the visual word ``water'' in image 4.

Our model can generally achieve good results by capturing the global semantics of an image sequence such as the example in the first row of Figure~\ref{fig:sis_caption}. However, our model also has difficulties in generating meaningful descriptions in a number of cases. For instance, our model generates fairly abstractive descriptions such as ``a great time'' due to severe overfitting, as shown in the second row of Figure~\ref{fig:sis_caption}. We suppose the issue of overfitting is likely to be alleviated by adding more training data or using more effective algorithm for image feature extraction.

\section{Conclusion}

In this paper, we present a local-object attention model with global semantic context for sequential image descriptions. Unlike other CNN-LSTM models that only employ a single image as input for image caption, our proposed method can generate descriptions of sequential images by exploiting the global semantic context to learn the dependencies between sequential images. Extensive experiments on two image datasets (DII and SIS) show promising results of our model.

\section*{Acknowledgement}
We thank Ehud Reiter and  Kees van Deemter for their helpful discussion. This paper was also supported partly by Science and Technology Project of Guangdong Province (No.503314759024).