text:
\section{Introduction}
Over the last decade,
an increasing number of processes are being automated through the use of
machine learning (ML) algorithms, being essential that these are robust and
reliable, if we are to trust operations based on their output. State-of-the-art
ML algorithms perform extraordinarily well on standard data, but recently they
have been shown to be vulnerable to adversarial examples, data
instances specifically targeted at fooling
algorithms \cite{goodfellow2014explaining}. As a
fundamental underlying hypothesis, these ML developments rely on the use
of independent and identically distributed data for both the training and test
phases, \cite{CARBONELL19891}. However, security aspects in ML, which form part of the
field of adversarial machine learning (AML), questions the previous
iid data hypothesis given the presence of adaptive adversaries ready to
modify the data to obtain a benefit; consequently, the
training and test distributions phases might differ.
Therefore, as reviewed in \cite{Albrecht2018AutonomousAM}, there is a need to model
the actions of other agents.

Stemming from the pioneering work in adversarial classification (AC) in
\cite{dalvi2004adversarial}, the prevailing paradigm used to model the confrontation
between adversaries and learning-based systems in AML has been  game theory,
\cite{menache2011network}, see the recent reviews
\cite{BIGGIO2018317} and \cite{doi:10.1002/widm.1259}. This entails
well-known common
knowledge hypothesis, \cite{hargreaves2004game}, which, from
a fundamental point of view, are  not sustainable in
security applications
as adversaries tend to hide and conceal information.
In recent work \cite{naveiro2018adversarial}, we have presented a novel
framework for AC based on Adversarial Risk Analysis (ARA) \cite{rios2009adversarial}.
ARA provides one-sided prescriptive
support to an agent, maximizing her expected utility, treating the
adversaries' decisions as random variables. To forecast them, we model
the adversaries' problems. However, our uncertainty about their probabilities
and utilities is propagated leading to the corresponding random optimal
adversarial decisions which provide the required forecasting distributions.
ARA makes operational the Bayesian approach to games, as presented in \cite{kadane1982subjective} or \cite{raiffa1982art}, facilitating a procedure to
predict adversarial decisions.

The AML literature has predominantly focused on
the supervised setting \cite{BIGGIO2018317}.
Our focus in this paper will be in reinforcement learning (RL).
In it, an agent takes actions
sequentially to maximize some cumulative reward (utility), learning from
interactions with the environment. With the advent of deep learning, deep RL
has faced an incredible growth \cite{mnih2015human, silver2017mastering}.
However, the corresponding systems may be also targets of
adversarial attacks \cite{huang2017adversarial,lin2017tactics}
and robust learning methods are thus
needed.
A related field of interest is multi-agent RL \cite{marl_over}, in which multiple
agents try to learn to compete or cooperate. Single-agent RL methods fail in
these settings, since they do not take into account the non-stationarity arising
from the actions of the other agents. Thus, opponent modelling is a
recent research trend in RL, see the Section \ref{sec:background} for a review of related literature.
Here we study how the ARA framework may shed light in security
RL and develop new algorithms.

One of the main contributions of the paper is that of introducing TMDPs, a framework to model adversaries that interfere with the reward generating processes in RL scenarios, focusing in supporting a specific agent in its decision making process. In addition, we provide several strategies to learn an opponent's policy, including a level-$k$ thinking scheme and a model averaging algorithm to update the most likely adversary. To showcase its generality, we evaluate our framework in diverse scenarios such as matrix games, a security gridworld game and security resource allocation games. We also show how our framework compares favourably versus a state of the art algorithm, WoLF-PHC. Unlike previous works in multi-agent RL that focus on a particular aspect of learning, we propose a general framework and provide extensive empirical evidence of its performance.

\section{Background and Related Work}\label{sec:background}

Our focus will be on RL. It has been widely studied as an efficient computational approach to \textit{Markov decision processes} (MDP) \cite{howard:dp}, which provide
a framework for modelling a single agent making decisions
while interacting within an environment. We shall refer to this agent as the
decision maker (DM, she).
More precisely, a MDP consists of a
tuple $\left( \mathcal{S}, \mathcal{A}, \mathcal{T}, R\right)$
where $\mathcal{S}$ is the state space with states $s$; $\mathcal{A}$ denotes the set of
actions $a$ available to the DM;
$\mathcal{T}: \mathcal{S} \times \mathcal{A} \rightarrow \Delta (\mathcal{S})$
is the transition distribution, where $\Delta(X)$ denotes the set of
all distributions over the set $X$;
and, finally, $R : \mathcal{S} \times \mathcal{A}  \rightarrow \Delta(\mathbb{R}) $
is the reward distribution modelling the utility that the agent perceives
from state $s$ and action $a$.

In such framework, the aim of the agent
is to maximize the long term discounted expected utility
\begin{equation}\label{kkk}
 \mathbb{E}_{\tau} \left[  \sum_{t=0}^\infty \gamma^t R(a_t, s_t) \right]
 \end{equation}
where $\gamma \in (0,1)$ is the discount factor and $\tau = (s_0, a_0, s_1, a_1, \ldots)$ is a trajectory of states and actions.
 The DM chooses actions according to a policy $\pi: \mathcal{S} \rightarrow \Delta(\mathcal{A})$, her objective being to find a policy maximizing
 (\ref{kkk}).

 An efficient approach to solving MDPs is Q-learning, \cite{sutton2012reinforcement}.
With it, the DM maintains a function
$Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ that estimates
her expected cumulative reward,
iterating according to the update rule
\begin{align}\label{eq:ql}
Q(s,a) &:= (1 - \alpha) Q(s, a)  +  \alpha \left(r(s,a) + \gamma\max_{a'} Q(s', a') \right),
\end{align}
where $\alpha$ is a learning rate hyperparameter
and $s'$ designates the state that the agent arrives at,
after having chosen action $a$ in state $s$ and received reward $r(s,a)$.

We focus on the problem of prescribing decisions to a single agent
in a non stationary environment as a result of the presence of other learning agents
that interfere with her reward process.
In such settings, Q-learning may lead
to suboptimal results, \cite{marl_over}.
Thus, in order to support the agent,
we must be able to reason about and forecast the adversaries' behaviour.
Several modelling methods have been proposed in the AI literature,
as thoroughly reviewed in \cite{Albrecht2018AutonomousAM}.
We are concerned with the ability of our agent to predict
her adversary's actions, so we restrict our attention to methods
with this goal, covering three main general approaches: policy reconstruction, type-based reasoning and recursive learning.
Methods in the first group fully reconstruct the adversary's decision
making problem, generally assuming some parametric model and fitting
its parameters after observing the adversary's behaviour. A dominant approach,
known as fictitious play \cite{brown1951iterative}, consists of modelling
the other agents by computing the frequencies of choosing various actions.
As learning full adversarial models could be computationally demanding,
type-based reasoning methods assume that the modelled agent belongs
to one of several fully specified types, unknown a priori, and learn
a distribution over the types; these methods may not reproduce the actual
adversary behaviour as none of them explicitly includes the ability of
other agents to reason, in turn, about its opponents' decision making.
Using explicit representations of the other agents' beliefs about their
opponents could lead to an infinite hierarchy of decision making problems,
as illustrated in \cite{rios2012adversarial} in a much simpler class of problems. Level-$k$ thinking approaches \cite{stahl1994experimental} typically stop this potentially infinite regress at a level in which no more information is available, fixing the action prediction at that depth through some non-informative probability distribution.

These modelling tools have been widely used in AI research but, to the best
of our knowledge, their application to Q-learning in multi-agent settings remains
largely unexplored. Relevant extensions
have rather focused on modelling the whole system
through Markov games, instead of
considering a single DM's point of view, as we do here. The three better-known solutions include minimax-Q learning \cite{littman1994markov}, where at each iteration a minimax
problem needs to be solved; Nash-Q learning \cite{hu2003nash},
which generalizes the previous algorithm to the non-zero sum case;
or the friend-or-foe-Q learning \cite{littman2001friend}, in which the DM knows in advance whether her opponent is an adversary or a collaborator. Within the bandit literature, \cite{auer1995gambling} introduced a non-stationary setting in which the reward process is affected by an adversary. Our framework departs from that work since we explicitly model the opponent through several strategies described below.

The framework we propose is related to the work of
\cite{lanctot2017unified}, in which the authors propose a deep
cognitive hierarchy as an approximation to a response oracle
to obtain more robust policies. We instead draw on the
level-$k$ thinking tradition, building opponent models that help the DM predict
their behaviour. In addition, we address the issue
of choosing between different cognition levels for the opponent. Our work
is also related with the iPOMDP framework in \cite{gmytrasiewicz2005framework}, though the authors only address the planning problem, whereas we are interested in the learning problem as well. In addition, we introduce a more simplified algorithmic apparatus
that performs well in our problems of interest. The work of \cite{he2016opponent} also addressed the setting of modelling opponents in deep RL scenarios; however, the authors rely on using a particular neural network architecture over the deep Q-network model. Instead, we tackle the problem without assuming a neural network architecture for the player policies, though our proposed scheme can be adapted to that setting. The work  \cite{foerster2018learning} adopted a similar experimental setting to ours, but their methods apply mostly when both players get to exactly know their opponents' policy parameters (or a maximum-likelihood estimator of them), whereas our work builds upon estimating the opponents' Q-function and does not require direct knowledge of the opponent's internal parameters.

In summary, all of the proposed multi-agent Q-learning extensions are inspired by game theory
with the entailed common knowledge assumptions \cite{hargreaves2004game}, which are not realistic in the
security domains of interest to us. To mitigate this assumption, we consider the problem of
prescribing decisions to a single agent versus her opponents,
augmenting the MDP to account for potential adversaries  conveniently modifying the Q-learning rule. This will enable us to adapt some of the previously
reviewed modelling techniques to the Q-learning framework, explicitly accounting
for the possible lack of information about the modelled opponent.
In particular, we propose here to extend Q-learning from an ARA \cite{rios2009adversarial} perspective, through a level-$k$
scheme \cite{stahl1994experimental, stahl1995players}.

\section{Threatened Markov Decision Processes}

In similar spirit to other reformulations of MDPs, such as
Constrained Markov Decision Processes \cite{altman1999constrained}
(in which restrictions along state trajectories are considered)
or Configurable Markov Decision Processes
\cite{2018arXiv180605415M} (in which the DM is able to modify the
environment dynamics to accelerate learning),
we propose an augmentation of a MDP
to account for the presence of adversaries which perform their
actions modifying state and reward dynamics, thus making the environment non-stationary.
In this paper, we mainly focus on the case of a DM (agent $A$, she) facing a single opponent ($B$, he). However, we provide an extension to a setting with multiple adversaries in Section \ref{sec:mul}.

\begin{definition}
A \emph{Threatened Markov Decision Process} (TMDP) is a tuple
$\left( \mathcal{S}, \mathcal{A}, \mathcal{B}, \mathcal{T}, R, p_A \right) $
in which $\mathcal{S}$ is the state space; $\mathcal{A}$ denotes the set of
actions available to the supported agent $A$; $\mathcal{B}$ designates the set of
threat actions $b$, or actions
available to the adversary $B$;
$\mathcal{T}: \mathcal{S} \times \mathcal{A}
\times \mathcal{B} \rightarrow \Delta(\mathcal{S})$ is the transition distribution;
$R : \mathcal{S} \times \mathcal{A} \times \mathcal{B} \rightarrow
\Delta(\mathbb{R}) $ is the reward distribution, the utility that
the agent perceives from a given state $s$ and a pair
$(a,b)$ of actions; and $p_A (b | s)$ models the DM's beliefs about
her opponent's move, i.e., a distribution over the threats for each state $s \in \mathcal{S}$.
\end{definition}
\noindent
 In our approach to TMDPs, we modify the standard Q-learning update rule (\ref{eq:ql}) by averaging over
 the likely actions of the adversary. This way the DM may anticipate potential threats within her
 decision making process, and enhance the robustness of her decision making policy. Formally, we first replace (\ref{eq:ql})  by
\begin{align}\label{eq:lr}
\begin{split}
Q(s, a, b) &:= (1 - \alpha)Q(s, a, b) + \\ &+ \alpha \left( r(s,a,b)
+ \gamma \max_{a'} \mathbb{E}_{p_A(b|s')} \left[ Q(s',a',b)  \right]  \right)
\end{split}
\end{align}
where $s'$ is the state reached after the DM and her adversary, respectively, adopt actions $a$ and $b$ from state $s$. We then compute its expectation over the opponent's action argument \begin{align}\label{eq:lr2}
Q(s,a) := \mathbb{E}_{p_A(b|s)} \left[ Q(s,a,b) \right].
\end{align}
We use the proposed rule to compute an $\epsilon-$greedy policy for the DM, when she is at state $s$, i.e., choose with probability $(1-\epsilon)$ the action
$a^* = \argmax_a  \left[ Q(s,a)  \right] $ or a uniformly random action
with probability $\epsilon$. \ref{sec:p} provides a proof of the convergence
of the rule. Though in the experiments we focus on such $\epsilon-$greedy strategy, other sampling methods can be straightforwardly used, such as a softmax policy (see \ref{s:softmax} for an illustration) to learn mixed strategies.

As we do not assume common knowledge,
the agent will have uncertainty regarding the
adversary's policy modelled by $p_A (b | s)$. However, we make the
standard assumption within multi-agent RL that both agents observe their opponent's actions (resp. rewards) after they have committed to them (resp. received them).
We propose two approaches to estimating the opponent's policy $p_A (b | s)$ under such assumption.
We start with a case in which the adversary is considered non-strategic,
that is, he acts without awareness of the DM, and then provide a level-$k$ scheme. After that, in Section \ref{sec:com} we outline a method to combine different opponent models, so we can deal with mixed behavior.

\subsection{Non-strategic opponent}\label{sec:non}

Consider first a stateless setting. In such case,
the Q-function (\ref{eq:lr}) may be written
 as $Q(a_i,b_j)$, with $a_i \in \mathcal{A}$ the action chosen by
 the DM and $b_j \in \mathcal{B}$ the action chosen by the adversary, assuming that $\mathcal{A}$ and $\mathcal{B}$ are discrete action spaces.
 Suppose the DM  observes her opponent's actions after he has
 implemented them. She needs to predict the action
 $b_j$ chosen by her opponent. A typical option is to model her adversary
 using an approach inspired by fictitious play (FP), \cite{brown1951iterative}: she may compute the expected utility of action $a_i$ using the stateless version of \eqref{eq:lr2}
\[ \psi(a_i) = \mathbb{E}_{p_A(b)} [Q(a,b)] = \sum_{b_j \in \mathcal{B}} Q(a_i, b_j) p_{A}(b_j), \]
where $p_A (b_j)$ reflects $A$'s beliefs about her opponent's actions
and is computed using the empirical frequencies of the opponent past plays, with $Q(a_i, b_j)$ updated according to (a stateless version of) \eqref{eq:lr}.
Then, she chooses the action $a_i \in \mathcal{A}$
maximizing her expected utility $\psi(a_i)$. We refer to this variant as
FPQ-learning. Observe that while the previous approach is clearly inspired by FP, it is not the same scheme, since only one of the players (the DM) is using it to model her opponent, as opposed to the standard FP algorithm in which all players adopt it.

As described in \cite{rios1}, adapting ARA to a game setting requires re-framing FPQ-learning from a Bayesian perspective. Let $p_j = p_B(b_j)$ be the probability with which
the opponent chooses action $b_j$. We may place a Dirichlet prior
$(p_1 , \ldots, p_n) \sim \mathcal{D}(\alpha_1,\ldots,\alpha_n)$,
where $n$ is the number of actions available to the opponent.
Then, the posterior is
$\mathcal{D}(\alpha_1 + h_1,\ldots,\alpha_n + h_n)$, with  $h_i$
being the count of action $b_i$, $i=1,...,n$.
If we denote its posterior density function as $f(p|h)$, the DM would choose the action $a_i$ maximizing her expected utility,
which adopts the form
\begin{eqnarray*}
& \psi(a_i) & = \int \left[\sum_{b_j \in \mathcal{B}}
Q(a_i, b_j) p_j\right] f(p|h) dp \\
&=& \sum_{b_j \in \mathcal{B}} Q(a_i, b_j) \mathbb{E}_{p|h}[p_j]
\propto  \sum_{b_j \in \mathcal{B}} Q(a_i, b_j) (\alpha_j + h_j).
\end{eqnarray*}
The Bayesian perspective may benefit the convergence of FPQ-learning,
as we include prior information about the adversary behavior when relevant.

Generalizing the previous approach to account for states is straightforward.
The Q-function has now the form $Q(s, a_i, b_j)$. The DM needs to assess probabilities $p_A(b_j | s)$,
since it is natural to expect that her opponent behaves differently depending
on the state.
The supported DM will choose her action at state $s$ by maximizing
\[ \psi_s(a_i) = \sum_{b_j \in \mathcal{B}} Q(s, a_i, b_j) p_{A}(b_j|s).  \]
Since $\mathcal{S}$ may be huge, even continuous, keeping track of $p_A(b_j|s)$ may incur in prohibitive memory costs. To mitigate this, we take advantage of Bayes rule as $p_{A}(b_j| s) \,\, \propto \,\, p(s| b_j)p(b_j)$. The authors of \cite{tang2017exploration} proposed an efficient method for keeping track of $p(s| b_j)$, using a hash table or approximate variants, such as the bloom filter, to maintain a count of the number of times that an agent visits each state; this is only used in the context of single-agent RL to assist for a better exploration of the environment. We propose here to keep track of $n$ bloom filters, one for each
distribution $p(s|b_j)$, for tractable computation of the opponent's intentions
in the TMDP setting. This scheme may be transparently integrated
within the Bayesian paradigm, as we only need to store an additional array with the Dirichlet prior parameters $\alpha_i$, $i=1,\ldots, n$ for the $p(b_j)$ part. Note that, potentially, we could store initial pseudocount as priors for each $b_j | s$ initializing the bloom filters with the corresponding parameter values.

As a final comment, if we assume the opponent to have memory of the previous stage actions, we could straightforwardly extend the above scheme using the concept of
mixtures of Markov chains \cite{raftery1985model},
thus avoiding an exponential growth in
the number of required parameters and linearly controlling
model complexity.
For example,
in case the opponent belief model
is $p_{A}(b_t | a_{t-1}, b_{t-1}, s_t)$, so that the adversary recalls
 the previous actions $a_{t-1}$ and $b_{t-1}$, we
 could factor it through a mixture
\[
p_{A}(b_t | a_{t-1}, b_{t-1}, s_t) = w_1 p_{A}(b_t | a_{t-1})  + w_2 p_{A}(b_t | b_{t-1})
 + w_3 p_{A}(b_t | s_t),
 \]
 with $\sum_i w_i = 1, w_i \geq 0, i=1,2,3$.

\subsection{Level-$k$ thinking}\label{sec:k}
The previous section described how to model a non-strategic (level-0) opponent.
This can be relevant in several scenarios. However, if the opponent is
 strategic, he may model our supported DM as a level-0 thinker, thus making
 him a level-1 thinker. This chain can go up to infinity, so we will
 have to deal with modelling the opponent as a level-$k$ thinker, with $k$
 bounded by the computational or cognitive resources of the DM.

To deal with it, we introduce a hierarchy of TMDPs in which
 $\emph{TMDP}_{i}^k$ refers to the TMDP that agent $i$ needs to optimize,
 while considering its rival as a level-$(k-1)$ thinker.
Thus, we have the process:

\begin{itemize}
\item If the supported DM is a level-1 thinker, she optimizes  $ \emph{TMDP}_{A}^1 $. She models $B$ as a level-0 thinker
(using Section \ref{sec:non}).
\item If she is a level-2 thinker, the DM optimizes
$ \emph{TMDP}_{A}^2 $ and models $B$ as a level-1 thinker. Consequently,
this ``modelled" $B$ optimizes $ \emph{TMDP}_{B}^1 $, and while doing so,
he models the DM as level-0
\item In general, we have a chain of TMDPs:
$$ \emph{TMDP}_{A}^k \rightarrow \emph{TMDP}_{B}^{k-1}
\rightarrow \emph{TMDP}_{A}^{k-2}  \rightarrow \cdots $$
\end{itemize}
Exploiting the fact that TMDPs correspond to repeated interaction settings
(and, by assumption, both agents observe all past
decisions and rewards), each agent may  estimate their
counterpart's Q-function, $\hat{Q}^{k-1}$:
if the DM is optimizing $\emph{TMDP}_A^k$, she will keep her own
Q-function (we refer to it as $Q_k$), and also an estimate
$\hat{Q}_{k-1}$ of her opponent's Q-function. This estimate may be
computed by optimizing $\emph{TMDP}_B^{k-1}$ and so on until $k=1$.
Finally, the top level DM's policy is given by
\[
\argmax_{a_{i_k}} Q_k(s, a_{i_k}, b_{j_{k-1}}),
\]
where $b_{j_{k-1}}$ is given by
\[
\argmax_{b_{j_{k-1}}} \hat{Q}_{k-1}(s, a_{i_{k-2}}, b_{j_{k-1}}),
\]
and so on, until we arrive at the induction basis (level-1)
in which the opponent may be modelled using the FPQ-learning approach
in Section \ref{sec:non}. Algorithm \ref{alg:l2ur} specifies the approach
for a level-2 DM.

Therefore,
we need to account for her Q-function, $Q_2$, and that of her
opponent (who will be level-1), $\hat{Q}_1$. Figure
\ref{fig:lev2_scheme} provides a schematic view of the dependencies.

\begin{algorithm*}[!ht]
\begin{algorithmic}
\Require $Q_2$, $\hat{Q}_1$, $\alpha_2, \alpha_1$ (DM and opponent Q-functions
and learning rates, respectively).
\State Observe  transition $(s, a, b, r_A, r_B, s')$
\State $\hat{Q}_1(s,b,a) := (1 - \alpha_1)\hat{Q}_1(s,b,a)  + \alpha_1 (r_B + \gamma \max_{b'} \mathbb{E}_{p_B(a'|s')} \left[ \hat{Q}_1(s',b', a') \right] )$
\State Compute $B$'s estimated $\epsilon-$greedy policy $p_A(b|s')$ from $\hat{Q}_1(s,b,a)$
\State $Q_2(s,a,b) := (1 - \alpha_2)Q_2(s,a,b) + \alpha_2 (r_A + \gamma \max_{a'} \mathbb{E}_{p_A(b'|s')} \left[ Q_2(s',a',b') \right]) $
\end{algorithmic}
\caption{Level-2 thinking update rule}
\label{alg:l2ur}
\end{algorithm*}
\noindent

\begin{figure}[!hb]
\centering
\stackinset{c}{.5in}{t}{.73in}{%
  \fboxrule=0pt\relax\framebox[2in][t]{%
  }}{\fboxrule=.75pt%
  \fbox{\stackunder{Level-2%
   \hspace*{\fill} (DM, denoted as A) }%
    {
    $Q_2$,
    $p_A(b | s) \leftsquigarrow$
    \fbox{\stackunder{Level-1 \hspace*{\fill} (Adv., denoted as B) }%
      {
      $\hat{Q}_1$,
    $p_B(a | s) \leftsquigarrow$
      \fbox{\stackunder{Level-0 \hspace*{\fill} (DM) }%
        {}
        }}}}}
}
\caption{Level-$k$ thinking scheme, with $k=2$}\label{fig:lev2_scheme}
\end{figure}

Note that in the previous hierarchy of policies the decisions are obtained
in a greedy manner, by maximizing the lower level
$\hat{Q}$ estimate. We may gain insight in a Bayesian fashion by adding
uncertainty to the policy at each level. For instance, at a certain level
in the hierarchy, we could consider $\epsilon-$greedy policies that with
probability $1-\epsilon$ choose an action according to the previous scheme
and, with probability $\epsilon$, select a random action. Thus, we may
impose distributions $p_k(\epsilon)$ at each level $k$ of the hierarchy.
The mean of $p_k(\epsilon)$ may be an increasing function with respect to
the level $k$ to account for the fact that uncertainty is higher
at the upper thinking levels.

\subsection{Combining Opponent Models}\label{sec:com}

We have discussed a hierarchy of opponent models.
In most situations the DM will not know which type of particular opponent she is facing.
To deal with this, she may place a prior $p(M_i)$ denoting her beliefs that her opponent is using a model $M_i$, for $i = 1, \ldots, m$, the range of models that might describe her adversary's behavior,
with $\sum_{i=0}^m p(M_i) = 1$.

As an example, she might place a Dirichlet prior. Then,
at each iteration, after having observed her opponent's action, she may
update her belief $p(M_i)$ by increasing the count of the model $M_i$ which
caused that action, as in the standard Dirichlet-Categorical Bayesian
update rule (Algorithm \ref{alg:update_averaging}). This is possible since
the DM maintains an estimate of the opponent's policy for each
opponent model $M_i$, denoted $p_{M_i}(b|s)$. If none of the opponent models had
predicted the observed action $b_t$, then we may not perform an
update (as stated in Algorithm \ref{alg:update_averaging} and done
in the experiments) or we could increase the count for all possible actions.

\begin{algorithm*}[!ht]
\begin{algorithmic}
\Require $p(M | H)\, \propto\, (n_1, n_2, \ldots, n_m)$, where $H$ is the sequence $(b_0, b_1, \ldots, b_{t-1})$ of past opponent actions.
\State Observe  transition $(s_{t}, a_t, b_t, r_{A,t}, r_{B,t}, s_{t+1})$.
\State For each opponent model $M_i$, set  $b^i$ to be the predicted action by model $M_i$.
\State If $b^i = b_t$ then update posterior:
$$
p(M | (H || b_t) ) \, \propto\,  (n_1, \ldots, n_i + 1, \ldots, n_m)
$$
\end{algorithmic}
\caption{Opponent average updating}
\label{alg:update_averaging}
\end{algorithm*}

Note that this model averaging scheme subsumes the framework of cognitive hierarchies \cite{camerer2004cognitive}, since a belief distribution is placed over the different levels of the hierarchy. However, it is more flexible since more kinds of opponents can be taken into account, for instance a minimax agent as exemplified in \cite{rios1}.

\subsection{Facing multiple opponents}\label{sec:mul}
TMDPs may be extended to the case of a DM facing more than one adversary. Then, the DM would have uncertainty about all of her opponents and she would need to average her Q-function over all likely actions of all adversaries. Let $p_A(b^1, \dots, b^M \vert s)$ represent the DM's beliefs about her $M$ adversaries' actions. The extension of the TMDP framework to multiple adversaries will require to account for all possible opponents in the DM's Q function taking the form $Q(s,a,b^1,\dots,b^M)$. Finally, the DM would need to average this $Q$ over $b^1, \dots, b^M$ in \eqref{eq:lr} and proceed as in \eqref{eq:lr2}.

In case the DM is facing non-strategic opponents, she could learn $p_A(b^1, \dots,$ $ b^M \vert s)$ in a Bayesian way, as explained in Section \ref{sec:non}. This would entail placing a Dirichlet prior on the $n^M$ dimensional vector of joint actions of all adversaries. However, keeping track of those probabilities may be unfeasible as the dimension scales exponentially with the number of opponents. The case  of conditionally independent adversaries turns out to be much simpler as we can use the fact that $p_A(b^1, \dots, b^M \vert s) = p_A(b^1 \vert s) \cdot \dots p_A(b^M \vert s)$. In this case, we could learn each $p_A(b^i \vert s)$ for $i=1, \dots, M$ separately, as explained in previous sections.

\subsection{Computational complexity}\label{sec:cc}

As described before, a level-$k$ Q-learner has to estimate the Q function
of a level-$(k-1)$ Q-learner, and so on. Assuming the original Q-learning
update rule has time complexity $\mathcal{O}(T(|\mathcal{A}|))$, with $T$ being a factor depending on the number of actions of the DM, the update rule from
Algorithm \ref{alg:l2ur} has time complexity $\mathcal{O}(kT(\max \lbrace |\mathcal{A}|,  |\mathcal{B}|\rbrace))$, i.e.,
linear in the level of the hierarchy.

Regarding space complexity, the overhead is also linear in the level $k$ since the DM only needs to store $k$ Q-functions, so the complexity is $\mathcal{O}(kM(|\mathcal{S}|,|\mathcal{A}| \cdot |\mathcal{B}|))$ with $M(|\mathcal{S}|,|\mathcal{A}| \cdot |\mathcal{B}|)$ accounting for the memory needed to store the Q-function in tabular form.

\section{Experiments and Results}

To illustrate the TMDP reasoning framework, we consider three
sets of experiments: repeated matrix games, with and without memory;
the adversarial security environment proposed in \cite{leike2017ai}; and a security resource allocation problem
in the form of a Blotto game. The first set of experiments allows us to illustrate several relevant computational
properties of our framework, whereas the second one illustrates relevant security games, our area of interest, and the
last set illustrate how our framework deals with structured action spaces and multiple adversaries.
All the code is released at \url{https://github.com/vicgalle/ARAMARL}, which includes full experimental setup details summarized in \ref{sec:exp_det}.

\subsection{Repeated Matrix Games}
We first consider experiments with agents without memory,
then agents with memory, and, finally, discuss general conclusions.
As initial baseline, we focus on the stateless version of a TMDP and
analyze the policies learnt by the DM, and analyze the policies learnt by the DM,
who will be the row player in the corresponding matrix game, against various kinds of opponents.
In all the iterated games, agent  $ i \in \lbrace A, B \rbrace $ aims at optimizing
$\sum_{t=0}^{\infty} \gamma^t r^i_{t}$, and we set
the discount factor $\gamma = 0.96$ for illustration purposes.
\subsubsection{Memoryless Repeated Matrix Games}\label{kk2}

We consider the classical Iterated Prisoner's Dilemma (IPD), \cite{axelrod84}. Table \ref{tab:payoffIPD} shows its reward bimatrix.
 Recall that, in this game, the Nash equilibrium is (D,D).

\begin{table}[h]
\begin{center}
\begin{tabular}{c|c|c}
\hline
 & C & D \\
\hline
C & (-1, -1) & (-3, 0) \\
\hline
D & (0, -3) & (-2, -2)  \\
\hline
\end{tabular}
\end{center}
\caption{Payoff Matrix of Prisoners' Dilemma}
\label{tab:payoffIPD}
\vspace{-2ex}
\end{table}

To start with, consider the opponent to be an independent-Q learner
(i.e., he uses the standard Q-function from single-agent RL and (\ref{eq:ql}) as learning rule). Fig. \ref{fig:IPD} depicts the utilities obtained over time by both players,
in cases where we model our DM as an independent Q-learner,
Fig. \ref{fig:QvsQ}, or as a FPQ-learner, Fig. \ref{fig:FPQvsQ}.
An opponent-unaware DM would remain exploitable by
another adversary (i.e., independent Q-learning does not converge
to the Nash equilibria).
Observe also that in Fig. \ref{fig:QvsQ} the variance is much bigger due to the inability of the basic Q-learning
solution to deal with a non-stationary environment.
In contrast, the level-1 FPQ-learner converges to the Nash equilibrium.
Indeed, the DM reaches the equilibrium strategy first, becoming stationary to her opponent,
and thus pulling him to play towards the equilibrium strategy.
Note that the FPQ-learner is unable to learn to
cooperate with her opponent, achieving lower rewards than her naive
counterpart. This is due to the specification of the environment
and not to a limitation of our framework since, as we
shall see in Section \ref{sec:mem}, the same agent with memory of past
actions is able to cooperate with its opponent,
when solving the previous problem.

\begin{figure*}%
\centering
\subfigure[Q-learner vs Q-learner]{%
  \label{fig:QvsQ}%
  \includegraphics[height=1.8in]{figures/QvsQ}}%
  \subfigure[FPQ-learner (blue) vs Q-learner (red)]{%
  \label{fig:FPQvsQ}%
  \includegraphics[height=1.8in]{figures/FPQvsQ}}%
  \caption{Rewards obtained in IPD. We plot the trajectories of 10 simulations with shaded colors. Darker curves depict mean rewards. }\label{fig:IPD}

 \subfigure[Q-learner vs Q-learner]{%
\label{fig:QvsQ_SH}%
\includegraphics[height=1.8in]{figures/QvsQ_SH}}%
\subfigure[FPQ-learner (blue) vs Q-learner (red)]{%
\label{fig:FPQvsQ_SH}%
\includegraphics[height=1.8in]{figures/FPQvsQ_SH}}%
\caption{Rewards in ISH game}\label{fig:ISH}

\subfigure[Q-learner vs Q-learner]{%
\label{fig:QvsQ_C}%
\includegraphics[height=1.8in]{figures/QvsQ_C}}%
\subfigure[FPQ-learner (blue) vs Q-learner (red)]{%
\label{fig:FPQvsQ_C}%
\includegraphics[height=1.8in]{figures/FPQvsQ_C}}%
\caption{Rewards in IC game}\label{fig:IC}

\end{figure*}

We turn to another social dilemma game, the Stag Hunt game, in which both agents must coordinate
to maximize their rewards. They payoff matrix is in Table \ref{tab:payoffSG}, with
two Nash equilibria (C,C) and (D,D). We designate its iterated version ISH. We use the same experimental setting as before and report results in Figure \ref{fig:ISH}.

\begin{table}[h!]
\begin{center}
\begin{tabular}{c|c|c}
\hline
 & C & D \\
\hline
C & (2, 2) & (0, 1) \\
\hline
D & (1, 0) & (1, 1)  \\
\hline
\end{tabular}
\end{center}
\caption{Payoff Matrix of Stag Hunt}
\label{tab:payoffSG}
\vspace{-2ex}
\end{table}
\noindent Once again, the independent learning solution cannot
cope with the non-stationarity of the environment and oscillates between both
equilibria  without clear convergence to one of
them (Fig. \ref{fig:QvsQ_SH}). On the other hand, the FPQ-learner converges quite rapidly to the socially optimal policy (Fig. \ref{fig:FPQvsQ_SH}). Then, the environment becomes
essentially stationary to its opponent, who also converges to that policy.

The last social dilemma that we consider is the Chicken game, with payoff matrix
in Table \ref{tab:payoffC}. It has two pure Nash equilibria (C, D) and (D,C). We designate its iterated variant by IC.

\begin{table}[h!]
\begin{center}
\begin{tabular}{c|c|c}
\hline
 & C & D \\
\hline
C & (0, 0) & (-2, 1) \\
\hline
D & (1, -2) & (-4, -4)  \\
\hline
\end{tabular}
\end{center}
\caption{Payoff Matrix of Chicken}
\label{tab:payoffC}
\vspace{-2ex}
\end{table}
\noindent
Figure \ref{fig:QvsQ_C} depicts
again the ill convergence due to lack of opponent awareness in the
independent Q-learning case; note that the instabilities continued cycling
even after the limit in the displayed graphics. Alternatively, the DM
with opponent modelling has an advantage and converges to her optimal Nash
equilibrium (D,C) (Fig. \ref{fig:FPQvsQ_C}).

\begin{figure*}[!ht]%
\centering
\subfigure[FPQ-learner (blue) vs WoLF-learner (red)]{%
  \label{fig:L1vsWoLF_C}%
  \includegraphics[height=1.8in]{figures/L1vsWoLF_C}}%
  \subfigure[L2Q-learner (blue) vs WoLF-learner (red)]{%
  \label{fig:L2vsWoLF_C}%
  \includegraphics[height=1.8in]{figures/L2vsWoLF_C}}%
  \caption{Rewards obtained in the IC game against a WoLF-PHC adversary. }
\end{figure*}

{\color{black}
In addition, we study another kind of opponent to show how our framework
can adapt to it. We consider an adversary that learns according to the
WoLF-PHC algorithm \cite{bowling2001rational}, one of the best learning approaches in the multi-agent reinforcement learning literature. Figure \ref{fig:L1vsWoLF_C}
depicts a FPQ-learner (level-$1$) against this adversary, where the latter
clearly exploits the former. However, if we go up in the level-$k$ hierarchy and model our
DM as a level-$2$ Q-learner, she outperforms her
opponent (Fig. \ref{fig:L2vsWoLF_C}).}

\subsubsection{Repeated Matrix Games With Memory}\label{sec:mem}

Section \ref{kk2} illustrated the ability of the modelled agents
to effectively learn Nash equilibrium strategies in several iterated games.
However, if we let agents have memory of previous movements, other types
of equilibria may emerge, including those in which agents cooperate. We can easily augment the agents to have memory of the past $T$ joint actions taken. However, \cite{press2012iterated} proved that, in the IPD, agents with a good memory-1 strategy can effectively force the iterated game to be played as memory-1, ignoring longer play histories. Thus, we resort to memory-1 iterated games.

We restrict our attention to the IPD. We model the memory-1 IPD as a TMDP in which the state $\mathcal{S}$ adopts the form $s_t = (a_{t-1}, b_{t-1}), \,\,  t > 0$
describing the previous joint action, plus the initial state $s_0$ in which there
is no prior action. Note that now the DM's policy is conditioned on
$\mathcal{S}$, and it is fully specified by the probabilities
$\pi(C | CC)$, $\pi(C | CD)$, $\pi(C | DC)$, $\pi(C | DD)$, and  $\pi(C | s_0)$.

\begin{figure}[h!]
\centering
\includegraphics[scale=0.5]{figures/MemvsTFT_G1}%
\caption{Rewards obtained by the DM for players: FPQ memoryless player vs TFT player (G1) and FPQ memory-1 player vs TFT player (G2).}\label{fig:Mem1}
\end{figure}

\noindent We assume a stationary adversary playing Tit-For-Tat (TFT), i.e.\ replicating
the opponent's previous action \cite{axelrod84}. TFT is a Nash equilibrium in the IPD with memory.
We aim at showing that the supported DM is able to learn the equilibrium strategy.

In the experiments, the adversary will compete with either an agent playing FP (the same from the stateless environment), or with a memory-1 agent also playing FP. Figure \ref{fig:Mem1} represents the utilities attained by these agents in both duels. As can be seen, a memoryless FPQ player cannot learn an optimal policy and forces the TFT agent to play defect. In contrast, augmenting this agent to have memory of the previous move allows him to learn the optimal policy (TFT), that is, he learns to cooperate, leading to a higher cumulative reward.

\subsubsection{Discussion}

We have shown through our examples some qualitative properties of the proposed framework. Explicitly modelling an opponent (as in the level-1 or FPQ-learner) is beneficial to maximize the rewards attained by the DM, as shown in the ISH and IC games. In both games, the DM obtains higher reward as a level-1 thinker than as a naive Q-learner against the same opponent. Also, going up in the hierarchy helps the DM to cope with more powerful opponents such as the WoLF-PHC algorithm.

In the two previous games, a level-1 DM makes both her and her opponent reach a Nash equilibrium, in contrast with the case in which the DM is a naive learner, where clear convergence is not assured. In both games there exist
two pure Nash equilibria, and the higher-level DM achieved the most profitable one for her, effectively exploiting her adversary.

The case of the IPD is specially interesting. Though the level-1 DM also converges to the unique Nash equilibrium (Fig. \ref{fig:FPQvsQ}), it obtains less reward than its  naive counterpart (Fig. \ref{fig:QvsQ}). Recall that the naive Q-learner
would remain exploitable by another opponent. We argue that the FPQ-learner did not learn to cooperate, and thus achieves lower rewards, due to the specification of the game and not as a limitation of our approach. To allow for the emergence of cooperation in the IPD, agents should remember past actions taken by all players. If we specify an environment in which agents recall the last pair of actions taken, the FPQ-learner is able to cooperate (Fig. \ref{fig:Mem1}) with an opponent that plays a Nash optimum strategy in this modified setting, Tit-For-Tat.

\subsection{AI Safety Gridworlds and Markov Security Games}\label{s:sg}

A suite of RL safety benchmarks has been recently introduced in \cite{leike2017ai}.
We focus on the safety \emph{friend or foe} environment, in which the supported DM
needs to travel a room and choose between two identical boxes, hiding positive
and negative rewards, respectively. The reward assignment is controlled by
an adaptive opponent. Figure \ref{fig:friendorfoe} shows the initial state in
this game. The blue cell depicts
the DM's initial state, gray cells represent the walls of the room.
Cells 1 and 2 depict the adversary's targets, who decides which one will
hide the positive reward.
This case may be interpreted as a spatial Stackelberg game in which
the adversary is planning to attack one of two targets, and the defender
will obtain a positive reward if she travels to the chosen target. Otherwise,
she will miss the attacker and will incur in a loss.

As shown in \cite{leike2017ai}, a \emph{deep Q-network}
(and, similarly, the independent tabular Q-learner as we show) fails to
achieve optimal results because the reward process is controlled by the adversary.
By explicitly modelling it, we actually
improve Q-learning methods achieving better rewards.
An alternative approach to security games in spatial domains was
introduced in \cite{klimamarkov}. The
authors extend the single-agent Q-learning
algorithm with an adversarial policy selection inspired by the EXP3 rule
from the \emph{adversarial multi-armed bandit}
framework in \cite{auer1995gambling}.
However, although robust, their approach does not explicitly model an adversary.
We demonstrate that by modelling an opponent the DM can achieve higher rewards.
\begin{figure}
   \centering
   \includegraphics[scale=0.5]{figures/friend-or-foeCUT}

   \caption{The \emph{friend or foe} environment from the AI Safety Gridworlds benchmark. Figure taken from \cite{leike2017ai}. } \label{fig:friendorfoe}
 \end{figure}

\subsubsection{Stateless Variant}\label{sec:statv}

We first consider a simplified environment with a singleton state and
two actions. In a spirit similar to \cite{leike2017ai}, the adaptive
opponent estimates the DM's actions using an exponential smoother.
Let $\bm{p} = (p_1, p_2)$ be the probabilities
with which the DM will, respectively, choose targets 1 or 2
as estimated by the opponent. At every iteration, the opponent updates his knowledge
through
$$
\bm{p} := \beta \bm{p} + (1 - \beta ) \bm{a},
$$
where $0 < \beta < 1$ is a learning rate, unknown from the DM's point of view,
and $\bm{a} \in \lbrace (1, 0), (0, 1) \rbrace$ is a one-hot encoded
vector indicating whether the DM  chose targets 1 or 2. We consider
an adversarial opponent which places the positive reward in target
$t = \argmin_i (\bm{p})_i$.
Initially, the opponent has estimates
$\bm{p} = (0.5, 0.5)$ of the target preferred by the DM.

Since the DM has to deal with a strategic adversary, we introduce a modification
to the FP-Q learning algorithm that places more attention to more recent actions. Leveraging the property that the Dirichlet distribution is a conjugate prior of the Categorical distribution, a modified update scheme is proposed in Algorithm \ref{alg:duwff}.
\begin{algorithm}
\begin{algorithmic}
\State Initialize pseudocounts $ \bm{\alpha^0} = (\alpha^0_1, \ldots, \alpha^0_n)$
\For{$t = 1, \ldots, T $}
\State $\bm{\alpha^t} = \lambda \bm{\alpha^{t-1}}$ \Comment Reweight with factor $0 < \lambda < 1$
\State Observe opponent action $b^t_i, i \in \lbrace b_1, \ldots, b_n \rbrace$
\State $\alpha^t_i = \alpha^{t-1}_i + 1$ \Comment Update posterior
\State $\alpha^t_{-i} = \alpha^{t-1}_{-i}$
\EndFor
\end{algorithmic}
\caption{Dirichlet updating with forget factor}
\label{alg:duwff}
\end{algorithm}
This approach essentially allows to account for the last $\frac{1}{1 - \lambda}$
 opponent actions, instead of weighting all observations equally.
For the case of a level-2 defender, as we do not know
the actual rewards of the adversary (who will be modelled as a level-1 learner),
 we  model it as in a zero-sum scenario, i.e. $r_B = -r_A$, making this case similar to the Matching Pennies game. Other reward
 scalings for $r_B$ have been considered, though they did not qualitatively affect
 the results (See \ref{sec:adv_rs}).

\begin{figure}%
\centering
\includegraphics[scale=0.55]{figures/4C_08_adversary}%
\caption{Rewards for the DM against the adversarial opponent}\label{fig:4C_adv}
\end{figure}
Results are displayed in Figure \ref{fig:4C_adv}. We considered three types
of defenders: an opponent unaware Q-learner, a level-1 DM with forget (Algorithm \ref{alg:duwff}) and a
level-2 agent. The first one is exploited by the adversary
achieving suboptimal results. In contrast, the level-1 DM with forget effectively
learns a stationary optimal policy (reward 0). Finally, the level-2 agent
learns to exploit the adaptive adversary achieving positive rewards.

Note that the actual adversary behaves differently from how the DM models him,
i.e. he is not exactly a level-1 Q-learner. Even so, modelling him as a level-1
agent gives the DM sufficient advantage.

\subsubsection{Facing more powerful adversaries}

Until now the DM has interacted against an exponential smoother
adversary, which may be exploited if the DM is a level-2 agent.
We study now the outcome of the process
if we consider more powerful adversaries.

First of all, we parameterize our opponent as a level-2 Q-learner, instead of the exponential smoother.
To do so, we specify the rewards he shall receive as $r_B = -r_A$, i.e., for simplicity we consider a zero-sum game,
yet our framework allows for the general-sum case. Figure \ref{fig:L2vsL2} depicts the rewards for both the DM (blue)
and the adversary (red). We have computed the frequency for choosing each action, and both players select either action
with probability $0.5 \pm 0.002$ along 10 different random seeds. Both agents achieve the Nash equilibrium, consisting of
choosing between both
actions with equal probabilities, leading to an expected cumulative reward of 0, as shown in the graph.

\begin{figure*}%
\centering
\subfigure[L2Q-learner (blue) vs L2Q-learner (red)]{%
  \label{fig:L2vsL2}%
  \includegraphics[height=1.8in]{figures/L2Q_vs_L2Q.png}}%
  \subfigure[L3Q-learner (blue) vs L2Q-learner (red)]{%
  \label{fig:L3vsL2}%
  \includegraphics[height=1.8in]{figures/L3Q_vs_L2Q.png}}%

  \subfigure[L3Q-learner (blue) vs L1Q-learner (red)]{%
  \label{fig:L3vsL1}%
  \includegraphics[height=1.8in]{figures/L3_vs_L1.png}}%
  \subfigure[L3Q-learner with opponent averaging (blue) vs L1Q-learner (red)]{%
  \label{fig:L3DirvsL1}%
  \includegraphics[height=1.8in]{figures/L3Dir_vs_L1.png}}%

  \subfigure[Estimate of  $P_{L1Q}$: DM's belief that her opponent is a level-1 Q-learner]{%
  \label{fig:probas}%
  \includegraphics[height=1.8in]{figures/probs.png}

  \caption{Rewards obtained against the exponential smoother adversary. }
\end{figure*}

Increasing the level of our DM to make her level-3, allows her to exploit a level-2 adversary, Fig. \ref{fig:L3vsL2}. However, this DM fails to exploit a level-1 opponent (i.e., a FPQ-learner), Fig. \ref{fig:L3vsL1}. The explanation to this apparent paradox is that the DM is modelling her opponent as a more powerful agent
than he actually is, so her model is inaccurate and leads to poor performance.
However, the previous ``failure" suggests a potential solution to the problem
using type-based reasoning, Section \ref{sec:com}. Figure \ref{fig:L3DirvsL1} depicts the rewards of a DM that keeps track of both level-1 and level-2 opponent models and learns, in a Bayesian manner, which one is she
actually facing. The DM keeps estimates of the probabilities $P_{L1Q}$ and $P_{L2Q}$ that her opponent is acting as if he was a level-1 or a level-2 Q-learner, respectively. Figure \ref{fig:probas} depicts the evolution of $P_{L1Q}$, and we can observe that it places most of the probability in the correct opponent type.

\subsubsection{Spatial Variant}

We now compare the independent Q-learner and a level-$2$ Q-learner against the
same adaptive opponent (exponential smoother) in the spatial gridworld domain,
see Fig. \ref{fig:friendorfoe}. Target rewards
are delayed until the DM arrives at one of the respective locations,
obtaining $\pm 50$ depending on the target chosen by the adversary.
Each step is penalized with a reward of -1 for the DM. Results are displayed
in Figure \ref{fig:4C_gridworld}. Once again, the independent Q-learner is
exploited by the adversary, obtaining even more negative rewards than in
Figure \ref{fig:4C_adv} due to the penalty taken at each step. In contrast,
the level-2 agent is able to approximately estimate the adversarial
behavior, modelling him as a level-1 agent, thus being able to obtain
positive rewards. Figure \ref{fig:L3Dir_spatial} depicts rewards of a DM that keeps opponent models for both level-1 and level-2 Q-learners. Note that although the adversary is of neither class, the DM achieves positive rewards,
suggesting that the framework is capable of generalizing between different model opponents.

\begin{figure*}%
\centering
\subfigure[Rewards for various DM models]{%
  \label{fig:4C_gridworld}%
  \includegraphics[height=1.8in]{figures/AI_safety}}%
  \subfigure[Rewards for a DM with opponent models for a L1 Q-learner and a L2 Q-learner (red)]{%
  \label{fig:L3Dir_spatial}%
  \includegraphics[height=1.8in]{figures/Level3Dir_vs_ExpSmoother.png}}%
  \caption{Rewards against the adversarial (exponential smoother) opponent in the spatial environment. }
\end{figure*}

\subsection{TMDPs for Security Resource Allocation}

We illustrate the multiple opponent concepts of Section \ref{sec:mul} introducing a novel suite of  resource allocation experiments which
are relevant in security settings. We propose a modified version of Blotto games \cite{hart2008discrete}: the DM needs to distribute limited
resources over several positions which are susceptible of being attacked. In the same way, each of the attackers has to choose different positions
where they can deploy their attacks. Associated with each of the attacked positions there is a positive (negative) reward of value 1 (-1). If the DM
deploys more resources than the attacks deployed in a particular position, she wins the positive reward; the negative reward will be equally divided between
the attackers that chose to attack that position. If the DM deploys less resources, she will receive
the negative reward and the positive one will be equally divided between the corresponding attackers.
In case of a draw in a given position, no player receives any reward.

We compare the performance of a FPQ-learning agent and a standard Q-learning agent, facing two independent opponents that are both using
exponential smoothing to estimate the probability of the DM placing a resource at each position, and implementing the attack where
this probability is the smallest (obviously both opponents perform exactly the same attacks). To that end, we consider
the problem of defending three different positions. The DM needs to allocate two resources among the different positions.
\begin{figure}%
\centering
\includegraphics[scale=0.55]{figures/2expsmoothers.png}%
\caption{Rewards for the DM against the adversarial opponent}\label{fig:2expsmoothers}
\end{figure}
As can be seen in Fig. \ref{fig:2expsmoothers}, the FPQ-learning is able to learn the opponents strategy and thus is less exploitable than the standard Q-learning agent. This experiment showcases the suitability of the framework to deal with multiple adversaries.

\section{Conclusions and Further Work}

We have introduced TMDPs, a novel reformulation of MDPs. This is an original
framework to support decision makers who confront opponents that interfere
with the reward generating process in RL settings. TMDPs aim at providing one-sided prescriptive support to a RL agent, maximizing her expected utility, taking into account potential negative actions adopted by an adversary. Some theoretical results are provided. In particular, we prove that our proposed learning rule is a contraction mapping so that we may use RL convergence results.
In addition, the proposed framework is suitable for using existing opponent modelling methods within Q-learning. Indeed, we propose a scheme to model adversarial behavior based on level-$k$ reasoning about opponents. We extend this approach using type-based reasoning to account for uncertainty about the opponent's level.

Empirical evidence is provided via extensive experiments, with encouraging results, namely the ability of the TMDP formalism to achieve Nash equilibria in repeated matrix games with an efficient computational scheme, as well as promote cooperation. In security settings, we provide empirical evidence that by explicitly modelling a finite set of adversaries via the opponent averaging scheme, the supported DM can take advantage of her actual opponent, even when he is not explicitly modelled by a component from the finite mixture. This highlights the ability of the proposed framework to generalize between different kinds of opponents. As a general lesson, we find that a level-2 Q-learner effectively deals with a wide class of adversaries. However, maintaining a mixture of different adversaries is necessary if we consider a level-3 DM. As a rule of thumb, the supported DM may start at a low level in the hierarchy, and switch to a level-up temporarily, to check if the obtained rewards are higher. Otherwise, she may continue on the initial, lower level.

Several lines of work are possible for further research. First of all, in the experiments, we have just considered up to level-3 DMs, though the extension to higher order adversaries is straightforward. In recent years Q-learning has benefited from advances from the deep learning community, with breakthroughs such as the
\emph{deep Q-network} (DQN) which achieved super-human performance in control
tasks such as the Atari games \cite{mnih2015human}, or as inner blocks inside
systems that play Go \cite{silver2017mastering}. Integrating these advances into
the TMDP setting is another possible research path. In particular, the
proposed Algorithm \ref{alg:l2ur} can be generalized to account for the use of deep Q-networks instead of tabular Q-learning as presented here. We show details of the modified algorithm in \ref{sec:deep}. Indeed, the proposed scheme is
\emph{model agnostic}, i.e., it does not matter if we represent the Q-function
using a look-up table or a deep neural network, so we expect it to be usable in both shallow and deep multi-agent RL settings.
In addition, there are several other ways to model the adversary's behavior that do not require to learn opponent Q-values, for instance by using policy gradient methods \cite{baxter2000direct}.
Finally, it might be interesting to explore similar expansions to semi-MDPs,
in order to perform hierarchical RL or to allow for time-dependent rewards
and transitions between states.

\vspace{2cm}
\textbf{Acknowledgements.}
VG acknowledges support from grant FPU16-05034,
RN acknowledges support from the Spanish Ministry for his grant FPU15-03636.
DRI is grateful to the MINCIU MTM2017-86875-C3-1-R project and the AXA-ICMAT Chair
in Adversarial Risk Analysis. All authors acknowledge support from
the Severo Ochoa Excellence Programme SEV-2015-0554. We are also very grateful to the numerous pointers and suggestions by the referees. This version of the manuscript was prepared while
the authors were visiting SAMSI within the Games and Decisions in Risk and Reliability program.
\pagebreak

redpajama_set_name: RedPajamaArXiv


Got a baby shower coming up? Is it the baby's first birthday? Buying a gift for a baby can be tricky, especially if you're not sure what the parents already have. Here are some tips for buying the perfect gift.
Be wary of buying newborn clothes. Newborn outfits may look adorable, but it's likely they may not even get worn. Parents-to-be are often inundated with newborn clothes. Some babies are born so big that none of these clothes fit from the start.
If you're buying baby clothes, consider something that can be worn a few months down the line. Parents often end up scrambling around for clothes at a later date and supplying some of these clothes for them could be a huge relief.
No noisy toys. Annoying noisy toys are a parents' worst nightmare. Unless you're getting back at them for doing the same to you, avoid electric mini-drumkits and soft toys that sing irritating tunes.
Personalize it. A personalized gift has some thought behind it and even if baby can't appreciate it, the parents will. You may be able to get a bodysuit online with an animal or country that means something to the parents. Alternatively, there are customised options such as monogrammed blankets and personalised soft toys. You may even have the arts and crafts skills to knit your own blanket or even craft your own teddy bear.
Think practical. If parents are strapped for cash, they may be in desperate need of some items such as a steriliser or baby monitor or extra diapers. Blankets and soft toys can be cute, but you could make more of an impact by buying something useful. Get some hints before a baby shower as to items they may need and offer to buy these for them.
Team together. Teaming together with fellow friends or family members may allow you to buy a big gift. This could be something important like a push-chair or a crib a car-seat. Such items often require a lot of saving up and the parents will appreciate you taking this load off their mind. As some mums can be fussy as to what pram or moses basket they want, it could be worth checking with them first as to which brands they're into. They may also know of deals that you could take advantage of to lower costs.
Donations are always appreciated. You may already have a crib or a baby bouncer that you can give as a present. This can be a kind gesture and will help out the parents a lot. You can even donate some money or give the parents some coupons along with another gift. This is sure to help along the way and can allow the parents to spend it on what they want.


text:
Q: Should I use MPM prefork or MPM event in apache? I am using Ubuntu 16.04 LTS with apache/2.4.18. I need to know which MPM module should I use whether MPM_prefork (which is default in my case) or MPM_event (which is a newly built module). It must be noted that we are dealing with huge traffic on webserver with configuration of 16-GB RAM and 8-cores CPU.

A: See this article for a good overview of the 3 main MPMs.
My suggestion would be to consider going with Event MPM as it should be better at dealing with high loads.

redpajama_set_name: RedPajamaStackExchange


ToyFinity has released the first photo of its Glyos-compatible Robo Force figures. An orange "test-shot" preview version of Maxx Steele will be available at NYCC this weekend. ToyFinity will also have three new colors of Mordles and the all-new jumbo-sized Mordle available (and all will be made available online later).
At Hasbro's "Twas the Night Before…" party on Wednesday night, they gave out free samples of Star Wars Black 6″ and Ron of ActionFigurePics got Han Solo. How I hate him. The good news is, I was worried about how this head sculpt would come out – the prototype seemed a little David Duchovny-y – but it seems to have turned out pretty well.
Here's an update from ToyGuru on Castle Grayskull.
Speaking of MOTUC, here are a couple more reveals – Flogg and what appears to be a battle-damaged, Terminator-style Faker head, probably part of a rumored "head pack." Check out the pics here.
Please welcome new PGPoA sponsor Nerd Rage Toys, who happen to have Star Wars Black 6″ available for pre-order.
Cool links: Dinosaur Dracula has a write-up on obscure Freddy Krueger merchandise and the giant, random Gorzak toy from the 1990s; Nerditis reviews the Heap from McFarlane Toys.
The MOTUC pix don;t seem to be working.
I got to see the Grayskull up close Friday, and for the first time almost feel like it's worth the cost. Almost. It's damned huge and would really need it's own piece of furniture to act as a pedestal.
I was very underwhelmed by the Matty panel. Seemed to me it announced very little new stuff. The Mini Masters were cool until I saw how much smaller than the originals they are.
I think Power-con, coming just a few weeks before NYCC, has stolen the thunder of the NYCC presentation. All the big reveals since SDCC show up at Power-con, leaving little for NYCC.
Considering this panel was less than a month after Powercon and only three months past SDCC, I thought it was pretty impressive. They announced one of the few remaining heroic warriors, the leader of the Space Mutants, teased a subscription for next year, announced a new line of minifigures that quite a few fans have been clamoring for (although the size issue is weird), and teased a long rumored head for Faker and a mystery accessory. Not bad, really- we know way more of next year's line than we ever have at this point before.
It is nice to see there is a good deal of new tooling on the 2014 MOTUC reveals. If the subs were still open I still wouldn't, as two-bad is the last figure I want, and it would have been nice to see more new tooling earlier on, but good for them for providing some much needed variety.
Pingback Castle Grayskull is Coming!


text:
Antarctic 48 hour film festival: Trusty Rusty
Open video: Antarctic 48 hour film festival: Trusty Rusty
G'day, Rusty here. Welcome to the first ever instalment of my adventure around this great white continent. Strap yourself in and let's go!
Today we're preparing for a journey to the Vestfold Hills in search of ancient relics and maybe even some long lost inhabitants. For what we're planning takes hours of meticulous planning, preparation and of course planning.
G'day mate.
Coming along on the journey with us this week is Chris, a young bloke from the city on his first trip to the continent.
I'll be showing him the ins and outs of Antarctic travel and making sure he doesn't get himself into too much trouble.
Now, I was just explaining to this young, inexperienced bloke here one can never underestimate the importance of securing your load securely with a series of FTO-approved knots. And as any FTO will tell you, if you can't tie knots, tie half inch lots.
Now naturally there's plenty of work to be done before we can take off. Let's load up these quads and head on out.
First stop will be over to the Met building to get a weather prediction for the next few days.
Now, whilst Davis station has an array of state-of-the-art weather sensing technology and equipment, of course nothing beats years of solid outdoor experience and a good old look up at the sky. But let's go.
A quick trip upstairs and we'll find our mate Jason hard at it.
G'day, Jase.
Rusty!
Now the Met folks work pretty hard to bring us the best predictions they can but they can't always be 100%.
But luckily for us, old Rusty knows exactly what he's doing.
And, ahh... so obviously this is your weather ensemble, yeah? For the pressure for the next couple of days?
Ah, nuh, Rusty. That's a drawing from my niece.
Let's not waste too much time though. We best be getting over to the workshop.
Here we'll find the station's best mechanics doing their finest work.
Let's hope they've got time to squeeze us in though.
Now the last thing we want in the field is any mechanical breakdowns.
That's why we've got Steve here giving everything the once over just to make sure it's tickety-boo.
Isn't that right, Steve?
Isn't that right, mate?
Now a quick step in to see the plumbers.
Alright. Oh, Jesus!
And then we'll be heading on in to comms to grab our radios for the trip.
Of course it pays to check your comms equipment before you take it out in the field because it's the only link you have back to the folks back here that can rescue you if needed.
Copy, Chris.
So, I was just telling Chris here the importance of navigating in the field and knowing where your north point is when you need it to get you out of a real bind.
So what you can do… is your trusty clock… a straight edge off any household object… so you're going to line up this with this… draw a line here to here… point that at the sun… and there you go… you're on your way.
Now it's time to be collecting one of the most important things for our trip.
Now to maintain energy in the field we need food and lots of it. Luckily for us, we've got our mate Rocket on the case to keep our bellies full. Let's go catch up.
Hey! G'day, Rocket.
Oh, what do you guys want?!
Just come to get a bit of food for our field trip.
More food! They always want food! Food! Here! Here! Food! I've had enough of youse! Get your food and get out!
Now, Rocket's always up for a chat.
But we best not waste any more time.
Now of course we don't expect to have any injuries in the field but if we do we need the right medical equipment so we'll go and see the doctor and get a first aid kit.
Hey, hey, get out!
Back down with you. We've got to find the source of this bleeding.
Now let's hope poor old Graham is soon on the mend.
As for us though, it's almost time for us to be heading off.
Now one last check of the equipment but I reckon I've got just about everything I need.
Your boots! Your clothes! Your motorcycle!
Nuyina construction update #2
Open video: Nuyina construction update #2
LEGO icebreaker sets sail on its maiden voyage
Open video: LEGO icebreaker sets sail on its maiden voyage
Hi! I'm Haidar Alnasser, one of the winners of the naming of the new icebreaker, Nuyina.
I'm standing here next to the new ship which is the new Lego model. One of my favourite features is the helicopter pad, where we can land and bring down cargo.
Another amazing part of this ship is the science deck right here where they can bring scientific samples and pass them off to the science laboratories we've got in the middle of the ship.
Across from the scientific laboratories we have the living quarters for all the scientists this is where they can relax, sleep, eat on their long journey down to Antarctica.
At the bottom of the ship here we have the massive engine room which powers all of the ship and helps it sailing along the way. Another amazing part of the ship is the moon pool down here - which is a big hole at the bottom of the ship - where scientists can lower down machines such as drones where they can capture all the sea life below. We've also got a really nice cool guy here dressed as a shark!
Now that I've shown you a few of the amazing features of this ship model, come check it out for yourself.
RSV Nuyina - Cabins
Open video: RSV Nuyina - Cabins
RSV Nuyina - Science operations
Open video: RSV Nuyina - Science operations
RSV Nuyina - Mess & Galley
Open video: RSV Nuyina - Mess & Galley
RSV Nuyina - Trawl deck
Open video: RSV Nuyina - Trawl deck
RSV Nuyina - Bridge
Open video: RSV Nuyina - Bridge
An aurora at Mawson
An aurora at Mawson station:
Open video: An aurora at Mawson
Sea ice, ocean swell, and ice shelf disintegration
Open video: Sea ice, ocean swell, and ice shelf disintegration
Barbara: Welcome everybody! Thank you so much for showing up for yet another exciting seminar. That reminds me, we won't have a seminar next week, and probably not the week after. So just watch out for emails; you'll get the invitation for probably, at the moment, for the 16 August.
Today's speaker is [Dr] Rob Massom, another well-known, happy face in the holy halls of science. Rob has been studying sea ice for no less than 35 years, which makes him an expert. He is currently the co-leader of the Sea Ice Process and Change program at the ACE-CRC. Way back in 1989, Rob did his PhD which was on data that he collected from drift buoys that were drifting around in the Weddell Sea, and Rob analysed the annual sea ice cycle based on these data. This was followed by a postdoc at the NASA Goddard Space Flight Centre, and in 1992 he finally made it to Tassie.
One of his many interests is the interaction between the Antarctic ice sheet and sea ice and that is what he will be talking about today. Thank you very much!
Rob: Thank you very much, Barb, for all those kind words. Thank you every one for coming to listen to me. Today I am going to talk about Surf's Up! It's about what happens - or what we think happens - to ice shelves when sea ice diminishes, as sea ice buffer around the ice shelves disappears I just like to start off by acknowledging my wonderful co-authors of this particular study; over there is Ted Scambos, who is an ice shelf and an ice sheet expert from the National Snow and Ice Data Centre from the University of Colorado. Next door to him is Luke Bennetts, who is a mathematician from the University of Adelaide who is an expert on sea ice and ice sheet - ocean interactions and modeling. And this character here who looks more like Amundson than a modern scientist is Professor Vernon Squire, who is also an expert on wave - ice interaction from the University of Otago. Close to home is Phil Reid from the Bureau of Meteorology, and also Sharon Stammerjohn from the University of Colorado. Both Phil and Sharon are experts on sea ice change and variability particularly change in duration which is a key factor when we are talking about ice shelves.
First of all, this is the structure of the talk; I'll say a little bit about why ice shelves are important in the scheme of things. Then we'll focus on five catastrophic disintegration events that occurred since 1995 involving the Wilkins, the Larsen A and the Larsen B ice shelves all of which are on the Antarctic Peninsula and we identified common factors that are involved in all of those, including disappearance of sea ice.
In a way this study is sort of breaking new ground - and I put there old ice as well - but it's breaking new ground in the sense that it's introducing sea ice change as a trigger mechanism that is involved in the disintegrations and that involves both pack ice, moving pack ice, and also stationary fast ice which is locked onto the periphery of these ice shelves. This has resulted in, culminated in a recently published article in Nature, about two weeks ago. I have to say that this is the culmination of 15 years of work. So thanks to my co-authors for helping me get this across the line.
Finally we'll wind up with a slide about what the future holds in terms of ice shelves and sea ice.
How, where and why Antarctica is losing mass is a very high priority and it's a major concern. Also two weeks ago, a there was a study in Nature by Shepherd et al. which had combined satellite measurements over the past 20 to 30 years, and they've shown - as in this plot here - that there has been a great acceleration in the loss of grounded ice from Antarctica to contribute to sea level rise into the ocean. They've shown that almost 2,700 billion tonnes of ice were lost since 1992 and that's the equivalent of about 8 mm of sea level rise. But more importantly, if you like, this loss is accelerating and 50% of the loss has occurred in the past 5 years. So this is looking quite serious.
The ice sheet contribution - I'm not an ice sheet scientist, I am very much a sea ice scientist - but the IPCC and all my colleagues tell me that the ice sheet contribution is the greatest uncertainty in predictions of future sea level rise. This is a quote from the IPCC down here. They say in the last report, the assessment report in 2013, that "due to a low level of scientific understanding there is low confidence in attributing the causes of the observed mass loss from the Antarctic Ice Sheet over the past two decades". Clearly we need to better understand what is causing this loss, what is driving it.
So what is at stake? The Antarctic Ice Sheet - as I am sure you know - holds roughly 90 % of all the ice on Earth and this is equivalent to - all melted in one go which isn't likely - but it's the equivalent of about 57metres of sea level rise. Importantly, a vast number of people live within 1 metre of mean high water level around the globe. If sea level rises again, you've got increased risk of storm surges affecting coastal margins.
So Ice shelf "health" - that's a hard one to say - is a key factor in all of this. Ice shelves and floating outlet glaciers fringe approximately three quarters of the grounded ice boundary of the whole of Antarctica. They play a key role; the role is to buttress or constrain the dynamic contribution, the flow of grounded ice down from the continent into the ocean past what's called the grounding line, which is this area over there. This is a very critical role in the sense that they constrain the amount of ice that comes down off the ice sheet and contributes to sea level rise. They do this by providing back pressure against outflow glaciers that are cascading down off the continent.
So if you remove this buttressing capacity, it's one of the ways and means of increasing the contribution of the ice sheet to sea level rise and a great deal of work has been done in the past decade or so looking at using satellite imagery in particular or satellite data to look at increased ice shelf thinning around the continent. You can see the red colours here show effectively where ice shelves have thinned due to incursions of warmer ocean waters on their bases. This map is from a study by Paolo et al. in 2015. Also there is a risk that the grounding line will retreat inwards and this could accelerate the amount of ice that comes off the continent. A lot of this work has been done in West Antarctica in the Pine Island glacier area. I just draw your attention to this position analysis from the ACE-CRC. It's well worth having a look at; it has recently been published by Gwyther et al. and it's "The Antarctic Ice Sheet and Sea Level"; it's a publication that pulls together all of that information.
It's critical, there is a critical need to better understand and indeed quantify the amount of melt, the patterns of melt, and the change that is occurring on the surface around Antarctica, but also how warm waters get underneath and interact with ice shelves to increase basal melt there.
A second way that more ice can come off the continent into the ocean is by disintegration of ice shelves, large scale disintegration and that's the focus of the talk here, indeed the study that was published. I show a sequence of images here that are showing the Larsen B; this is in late January 2002 just before it broke up and I draw your attention to melt ponds on the surface there, and also the fact that there is no sea ice in the ocean in the northwestern Weddell Sea just offshore from the ice shelf. Over just a very short period of time, in fact two weeks, the whole thing disintegrated. You would have seen these images; it was quite stunning the way this happened so rapidly. Over 3000 square kilometres were lost in just over 2 weeks and this was a feature that had been in place - based upon sedimentary measurements underneath the ice shelf once it disappeared - been in place for 11,500 years. So something that had been in place for a long period of time disintegrated literally overnight.
When ice shelves disintegrate, they don't directly contribute to sea level rise. The ice shelf is already floating. But in the case of this particular disintegration, the glacier flowing into it accelerated dramatically. There was an 8-fold acceleration of glaciers in this region here in the year following the disintegration of the ice shelf. This is based upon three studies, one of which is shown here, by Rignot et al. 2004, based again on satellite data analysis. That outlet glacier was also lowered by 38 metres over that time period.
Here we are going to concentrate on three ice shelves and five disintegrations events; it's the Wilkins ice shelf, which is here over on the south western part of the peninsula, and the Larsen A and Larsen B which are on the north eastern part of the peninsula. In each case, hundreds to thousands of square kilometres of ice disintegrated very, very rapidly in just a few days or even weeks. This was extraordinary, this abruptness and rapidity and the scale of the break-up was quite extraordinary, and marked a major departure from the typical calving scenario that occurs around Antarctica on decadal time scales, the kind of big ice bergs that break off naturally. Once these ice shelves broke up, there was again based on the Shepherd study, there was a five-fold increase in grounded ice loss from the Antarctic Peninsula since 1992. So the disintegration of these ice shelves played a big role in that. This is really the most special part of our study; understanding the causes of these disintegrations is crucial to improve the modeling of the ice shelf - ice sheet - ocean and atmosphere and here we add sea ice because we feel that this is a key part of the system as well, depending upon which ice shelves you are talking about. It is important, it is necessary to predict the fate of the remaining ice shelves in future and the fate of the ice sheet indeed as well, to reduce current large uncertainties in future estimates of the ice sheet contribution to sea level rise.
These disintegrations to date have been attributed to a couple of key things. Strong regional warming - as you are probably aware - the Antarctic Peninsula has been one of th strongest warming regions on Earth since measurements began in the 1940s. Where you get the warming, you get changes in material strength of the ice sheet, the ice shelves, and surface melt and brine infiltration as well. In addition to this, you get increased ponding on the surface as shown in this image of the Larsen B just before it broke up in January 2002. Now this increased ponding, as studies have shown, the water while it sits on the surface can actually infiltrate into existing crevasses and cracks or discontinuities within the ice shelf. The pressure of the water can force down and increase and open those cracks even more and compromise the structural integrity of the ice shelf. That has been called hydro-fracture. In addition to this warming, there are inherent glaciological factors involved as well. Outer margin fracture, the outer region here by bending stresses. This occurs where ice shelves melt and you get a redistribution of the stresses and that puts additional strain on the ice shelves and results in more cracking. You also get structural weaknesses which can occur inland where a glacier is cascading down from the ice sheet in contact with the bedrock or mountains and those rifts propagate down through time to affect the structural integrity of the ice shelf. Those two factors, strong regional warming and glaciological factors, appear to reduce the structural integrity or destabilize an ice shelf but they don't quite explain what we are seeing in terms of the break-up, that abrupt transition from a quasi-stable state to wide-scale disintegration. So what we are suggesting is could regional change in sea ice be an additional factor? And that is the story behind our paper.
In fact, all of the disintegrations to date - that's the Larsen A, the Larsen, and the Wilkins Ice Shelf over here - they have occurred adjacent to hot spots of decreased annual sea ice duration. All the reds in this map show where the sea ice annual duration has decreased. It's a trend and that is using satellite passive microwave data or ice concentration data. Where these disintegrations have occurred, the ice season, the amount of year when ice covers the ocean offshore is approximately three months shorter than it used to be. That is suggesting that these ice shelves have been exposed to open ocean and conditions more than they were in the past.
Every time I look at this I... I'm colour blind so you have to bear with me. I am trying to work through it as best as I can. Anyway, what these plots are showing is a time series of satellite derived ice concentration in boxes which are immediately offshore from the ice shelves that broke up. This is Larsen and this is Wilkins over here. What it's showing is this at the top here is 100 percent sea ice, so that is fully consolidated sea ice around the ice shelves. Down at the bottom is the opposite; it's open ocean around ice shelves. In both cases, you can see something of a change from more compact sea ice or more sea ice around the ice shelves to greater frequency of occurrence of open water periods around each ice shelf, particularly in the Wilkins but also in the Larsen here. The change seemed to occur roughly in 1991 for the Larsen and 1989 for the Wilkins. Looking at the annual values of these, we've shown that the Larsen B and A were exposed mainly - or more exposed - over a period from November to March, while the Wilkins was February to May and even June which is outside the melt season. That suggests to us there is something appearing here which is in addition to melt; hence, the sea ice loss. Previous studies have shown that this is in line with a study by Arigoney-Neto et al. in 2014 where they pooled together all the information on the Wilkins. They have shown that for the period 1947 to 1990, that indeed this Wilkins area didn't suffer any break-ups at all. But break-ups suddenly occurred once the sea ice started diminishing more and more.
What this plot is showing down here is particularly extreme event, an atmospheric event that occurred in January. The break-up occurred in January to March 2002 of Larsen B. This was related to an extreme exposure event, a wide open ocean corridor that existed from mid-November 2001 to mid-March 2002. This has been attributed to persistent northerly and north-westerly winds. I was stuck on an ice breaker with the Americans on the other side of the peninsula in the lead-up to this event. That's a different story but it produced a study there which showed the effect of compaction of sea ice and warm conditions on that side of the peninsula. What happened on the other side of the peninsula is that all these northerly winds swept the ice away and that is shown in the red colours in that bottom map. So it was total exposure of that ice shelf region.
So the first common factor is loss of sea ice; the second common factor appears to be this calving of outer - we call them sliver ice bergs; long, thin bergs - that occurred just off the margins of the ice shelves just prior to them disintegrating. You can see them here from the Larsen B and soon afterwards this mélange plume, the whole thing just disintegrated as a plume into the northwestern Weddell Sea. It's also the case with the Wilkins in 2008 and 2009, and also the Larsen A. So this let us to think are these two common factors - this lack of sea ice and this outer margin calving - are they somehow linked?
This was the basis of our paper and we put together this conceptual model. I am not expecting you to have a look at this and take it all in but what it is showing is on the left hand side that there are these ice shelf factors or glaciological factors that are part and parcel of these disintegrations but the ocean is playing a key role. The atmosphere is obviously playing a key role in terms of warming and the winds sweeping ice away. But what is new to this study is the sea ice factors over here. We are suggesting that sea ice loss somehow precipitated increased attack of these outer margins of the ice shelves, very vulnerable margins where with the waves and ocean swells and these triggered the disintegration of the ice shelves having first calved the sliver bergs.
Again I am not expecting you to take all of this in but the question is how does swell damage ice shelves in the absence of sea ice? Luckily, as I said we had some first rate modelling capability through Vernon in particular and Luke as well, of course. They looked at two aspects of this one of which -number 1 there - is the effect that sea ice has on in terms of damping or attenuating pressure waves that are coming towards the ice shelves. So we were able to look at the effects of sea ice and we fed into the models some information that we derived from the literature and also from satellite data. The second modelling is to do with the strain of the ice shelf itself due to the waves. We had a look at this strain with and without the sea ice cover around.
So what was the effect of this sea ice removal on the swell-induced strain? We were able to look - based upon wave model data - the peak periods we were looking at were in the vicinity of about 3 to 12 seconds for Larsen, and the Wilkins is 3 to 16 seconds, and significant wave height is about 3 to 7 metres and the directions were largely on-shelf when these disintegrations occurred. So with the sea ice buffer, there was strong attenuation of waves in these kind of ranges. That was shown by our model results and has also been shown by previous studies. The maximum strain that is due to this on the ice shelf itself were of the order of 10-8 to 10 -7. What these plots here are showing is here is a maximum strain at 10-5. The blue line is where there is no sea ice, and this area over here is including sea ice that is a sea ice zone which is about 80 km wide, and this is 250 km wide. So having sea ice makes a big difference to the amount of energy that can interact with the outer margins. When you take sea ice away, these strains increase significantly to about 10 -6. But for an 80 m thick ice shelf they increase to about 10 -5 and these kind of strains - based upon previous studies gain by Vernon Squire and colleagues way back into the 1970s - are sufficient to enlarge outer margin rifts and crevasses due to their fexural cycling and fatigue caused by these swells interacting with the outer margins. So 80 metres thick ice shelves seems very thin but studies on the Larsen C have shown that basal crevassing in the outer margin of the ice shelves quite dramatically affect the coherent thickness. Based upon those observations we suggest that the Larsen B and the Larsen A could be affected by the same processes whereby the basal crevasses decreased the thickness by approximately two thirds.
Another important thing is that the model showed us the plot over here; it's the maximum strain location in kilometres, from the outer margin in to the ice shelf and this shows the observed upper range of the waves, this is wave period down here. What this is showing us is that the maximum flexural strain occurred not deep into the ice shelf because all of the energy is dissipated by the thickness of the ice shelf itself. But it was concentrated on the very outer margins, and this is exactly where the sliver berg calving was occurring. It is also coincident with outer zones of increased margin fracturing. This is a satellite based image of the Larsen B before it broke up - that's the Larsen B there - and you can see this whole network of very important in this case crevasses that occurred towards the outer margins and indeed increased in length and number after January 2000 just before the ice shelf break-up. This is a similar feature in the Wilkins; this is the northern margin of the Wilkins and you can see the same structures which are barely holding together the ice shelf. Down in this area here, this is more protected by Latay Island from damaging swells and there has been less loss from that region just there.
There is further evidence of a relationship between sea ice change/variability and break-ups, in this case the Wilkins. As I said the break-ups of the Wilkins which were both catastrophic occurred not only in summer but also in winter and autumn. They coincide with increased exposure and low sea ice concentrations that occurred offshore. This is showing the 2008/2009, this is the seasonal mean sea ice concentration anomalies for each of the seasons again challenging my colour blindness but I had to write summer and autumn in there so I get it right. There were earlier break-ups here as well which coincide with these dips in sea ice coverage or sea ice anomalies around and certainly the case in 2008/2009 also.
We're on the home straight now! This is showing the Wilkins Ice Shelf; not only pack ice is the factor but an additional factor is land-fast or fast ice that is attached to the outer margins. You can see it in 2008 and 2009. A disintegration in 2008 and 2009 only occurred when this fast ice broke up and diminished. You can see here in 2008/2009 when there was fast ice, no break-up; when the fast ice broke up, the collapse quickly followed and it's even more dramatic in 2009. We think that fast ice plays an important role and acts as an additional buffer to destructive storm related swells. I should say we are looking at swells that come from distant storms across the global ocean. They also act as a mechanical bonding agent/buttress. They act as a glue if you like on the outer margins and minimise calving and also stabilise the ice shelves.
There is evidence from the Mertz Glacier Tongue of strong mechanical coupling between fast ice and ice shelves from a previous study. What you can see in this image here - it's a satellite synthetic aperture radar image from RadarSat - you can see that there is rifting in the glacier tongue there but this rifting was propagating through the fast ice which is attached to it and moving with it. That is suggesting that there is some mechanical coupling there and also we showed that the Mertz Glacier Tongue which was advancing at approximately 1 kilometre for the year - was advancing en masse with fast ice attached to it; this is shown in the relative displacement. We looked at the relative motion of the glacier and the fast ice using satellite imagery. This is just to show that on this side where there is fast ice, there is less calving, small scale calving, of ice bergs and on this side where there is a polynya there is significant calving of small scale ice bergs there.
This is showing that - based on additional work by Alex Fraser - this is a plot showing sea ice concentrations from that box region offshore from the Wilkins Ice Shelf. This is consolidated sea ice and open ocean just there; complete exposure down here. This is MODIS fast ice anomaly here; that is zero here, and negative and positive. What we are seeing here is that where there is extensive pack ice, there is also extensive fast ice; the pack ice protects the fast ice. In the years when there was break-up of the ice shelves, 2008/2009, there was persistent lack of pack ice offshore and this led to less fast ice and then break-ups. So pack ice effectively - this based on work that has been done over the years by Langhorne et al. and other people - shows that pack ice plays a key role in buttressing fast ice from swell break-up, something that anyone that studies fast ice, - penguin scientists like Barb or anyone else - understands this relationship. There is an intimate relationship; change in pack ice can affect fast ice, it affects - we suggest - ice sheet margin stability.
So, the synthesis, pulling all this together. We are thinking that in the cases we looked at, the Wilkins and the Larsen A and the Larsen B, a change in sea ice cover led to enhanced margin interaction of the ice shelves with swells. This over time cumulatively led to outer margin weakening to the point where these bergs, the sliver berg calvings, occurred. In effect, this removed key stone blocks from if you like an arch, relatively stable "compressive arch". This has been seen in satellite analysis that showed the strain rate trajectory is transvers when they are in place and the compressive arches are in place, then the ice shelf outer margin is relatively stable. But if you remove some blocks from that outer margin, it can lead to structural failure. Again basal crevassing would increase the likelihood of swell induced outer margin failure.
An earlier study by Doake et al. in 1998 predicted - and this is a modelling study - that outer margin failure, perhaps just a few kilometres inland, would trigger spontaneous chain reaction collapse of weakend ice shelves. This will be driven by stored potential energy of the system. A number of other studies have shown that this is water driving into cracks, hydrofracture. Once you set the thing in motion - and McAyeel and others have shown that this is ice blocks toppling like dominoes and moving outwards, and also localised tsunami-genesis where this toppling occurs that can generate significant waves which then feed back into the system to create further break-up. And another study by Banwell et al. has shown that chain-reaction drainage of the surface lakes can be another part of this chain reaction. The plume expansion - this whole plume of ice bergs - can extend out into the ocean if there is no sea ice buttress there. In the case of the Wilkins, this was unconstrained by the presence of fast ice. Also in the case of the Wilkins, fast ice plays a key role in reconstituting the ice shelf. So not always do the plumes blow out and remove all of the material. Some of it sits in place and is frozen together by fast ice. There are other studies that have shown there is a possibility of very long period waves affecting ice shelves. These are unaffected by sea ice presence or absence. Another possibility is that tidal effects can be playing a role but these are thought to be less significant and mainly occurring near the grounding line.
To wrap up: there are four common, essential prerequisites for disintegration in the cases that we have looked at, we think. The first is reduced coverage of sea ice and prolonged exposure to ocean swells. The second is extensive surface flooding and hyrofracture where the water goes down into crevasses. The third is extensive outer margin fracturing, and the fourth just before these disintegrations occur is that calving of sliver bergs. What we are suggesting is that - again in the cases that we looked at - while flooding and hydrofracture, which is the number 2 case there, are the drivers of catastrophic disintegration; it's the linkages between reduced sea ice, extensive margin fracturing and sliver berg calving which play a key role in not only preconditioning or destabelising ice shelves but also doing that to the point that they can trigger rapid disintegration of shelves weakened by multiple factors. Sea ice loss increases the odds of disintegration.
What does the future hold? IPCC and others have predicted that there is the rising temperature in the future in Antarctica. This could lead to weakening of ice shelves - that is remaining ice shelves, there is predicted sea ice loss and also more extreme wave states around the Southern Ocean. This could increase the likelihood of future ice shelf disintegrations. However, not all remaining ice shelves are likely to respond in the same way to such increased exposure. It also depends upon their glaciological characteristics, physical setting, and the degree and nature of flooding. This is a study [by Bell et al.] This image is of the Nansen Ice Shelf - it's by Bell et al. and came out a year or so ago; it shows that melt is not always destructive; sometimes - in this case - there are rivers or channels which move melt water off the surface into the ocean rather than the melt water sitting in ponds and creating fractures. Cold climate ice shelves with frequent open water offshore are apparently stable, we believe; examples being the Fimbul and Ross Ice Shelf. They can withstand wave flexure and this has been shown by a study by Bromirski et al. on the Ross Ice Shelf without sudden retreat IF they are not flooded and hydrofractured and the structural integrity is not compromised. But will this situation change in the future?
So final thoughts. The focus today has been largely on important ice shelf and ocean interactions regarding thinning. This is a very nice figure from the report by Gwyther et al. What we are suggesting is that sea ice change and variability could be an additional player to bring into this whole model regarding ice shelf stability and disintegration. This is in addition to the role of sea ice in modulating basal melt. A number of studies recently by Gwyther et al. and Silvano et al. and others have shown that where you get strong polynya activity, this puts a plume of salinity or saline water down into the water column which effectively cuts off incursions of warm water to lead to basal melt. This figure again, our study and other studies underline the highly coupled and complex nature of the ice shelf system which around Antarctica is undergoing really rapid change. It suggests that ice sheet models should include sea ice and waves as something of a step to forecasting the fate of the remaining shelves and reducing uncertainty in sea level rise. There are many unknowns; we need more understanding and modelling and observations on the linkages and the nature of these effects - mechanical or otherwise - and couplings. I guess the take away message here - I have to think hard about this - is sea ice change affects sea level rise. Sea ice is important on quite a number of levels; it's important for biology, the climate, biogeochemistry, etc. But this is another reason to better understand what is happening in sea ice, with sea ice around Antarctica. What is driving the observed trends, the patterns of change and variability? This is towards improved modelling and prediction and future coverage, and here we can add coastal exposure, too.
So follow up work in progress. We are developing through satellite remote sensing - Phil and myself and others - a Coastal Exposure Index. We are putting together a review paper on two-way linkages between sea ice and ice sheet margins and ice bergs, and that's both in the Arctic and Antarctic. The idea is to try and raise the profile of cross-cryosphere interactions/ linkages and identify gaps, and encourage inclusion in models, adopting a more holistic approach. With Alex and others, we'll be looking at case studies, regional, again of cross-cryosphere interactions and change involving pack ice, ice bergs, floating margins and fast ice.
I just like to wind up by acknowleding my co-authors and Wendy Piper and Nisha Harris who dealt very nicely - and I am very grateful - dealt nicely with the media around this about two weeks ago. This contributes to a WCRP CliC initiative and if anyone is interested, there is a Nature paper which is quite dense but there is also a conversation article which is hopefully a little bit more approachable if anyone is interested in following this up. And I always put this up and Go, the roos!

redpajama_set_name: RedPajamaCommonCrawl


C. S. Lewis told us that we have three options when it comes to evaluating the life of Jesus. He can either believe Him to be the Lord of the universe, a raving lunatic, or the world's greatest liar. There are no other options.
Jesus made some audacious claims. He claimed that He, alone, was the only way to God (John 14:6). He claimed that all who believe in Him will live forever (John 3:16). He claimed that He was going to be killed in Jerusalem and then rise from dead (Matthew 16:21).
You have three options with these claims. You can say that Jesus didn't really believe these things. Rather, He was a liar attempting to deceive others into following Him. Or, you can say that Jesus really believed them, but He was wrong. This would make Him to be a crazy lunatic. Or, you can say that Jesus believed these things and backed them up with his life, which makes Him Lord. There really is no other option.
So, let's be done with the idea of respecting Jesus because He was a really good teacher.