text:
\section{Introduction}

Many real-world applications (\emph{e.g.}, autonomous driving~[16], game playing~[12] and robotics control~[9]) today require reinforcement learning tasks to be carried out in multi-agent settings.
In MARL, multiple agents interact with each other in a shared environment. Each agent only has access to partial observations of the environment, and needs to make local decisions based on partial observations as well as both direct and indirect interactions with the other agents.
This complex interaction model has introduced numerous challenges for MARL. In particular, during the training phase, each agent may dynamically change its strategy, causing dynamics in the surrounding environment and instability in the training process. Worse still, each agent can easily overfit its strategy to the behaviours of other agents~[11], which seriously deteriorates the overall performance.

In the research literature, there have been three lines of research that try to mitigate the instability and inefficiency caused by decentralized execution. The most common approach is independent Q-learning (IQL)~[20], which breaks down a multi-agent learning problem into multiple independent single-agent learning problems, and makes each agent learn and act independently. Unfortunately, this approach does not account for instability caused by
environment dynamics, and therefore often leads to collided actions during the execution.
The second approach adopts the centralized training and decentralized execution~[18] paradigm, where a joint action value function is learned during the training phase to better coordinate the agents' behaviours. During execution, each agent acts independently without direct communication. The third approach introduces communication among agents during execution~[17,3]. This approach allows each agent to dynamically adjusts its strategy based on its local observation along with the information received from the other agents. Nonetheless, it introduces
additional communication overhead in terms of latency and bandwidth during execution, and its success is heavily dependent on the usefulness of the received information

In this work, we leverage the advantages of both the second and third approach.
Specifically, we consider a fully cooperative scenario where all the agents collaborate to achieve a common objective. The agents are trained in a centralized fashion within the multi-agent Q-learning framework, and are allowed
to communicate with each other during execution. However,
unlike previous work, we have a few key insights. First, for many applications, it is often superfluous for an agent to wait for feedback from all surrounding agents before making an action decision. For instance, when the front camera on a autonomous vehicle detects an obstacle within the dangerous distance limit, it triggers the `brake` signal without waiting for the feedback from the other parts of the vehicle. Second, the feedback received from the other agents may not always provide useful information. For example, the navigation system of the autonomous vehicle should pay more attention to the messages sent by the perception system (\emph{e.g.}, camera, radar), and less attention to the entertainment system inside the vehicle before taking its action. Full communication among the agents leads to a large communication overhead and latency, which is impractical for the real system implementation with strict latency requirement and bandwidth limit (e.g., real-time traffic signal control, autonomous driving, etc). In addition, as pointed out by~[7], excessive amount of communication may introduce useless and even harmful information which could even impair the convergence of learning.

Motivated by these observations, we design a novel deep MARL architecture that dramatically improves inter-agent communication efficiency.
Specifically,
we introduce Variance Based Control (VBC), a simple yet efficient approach to reduce the information transferred between the agents. By inserting an extra loss term on the variance of the exchanged information, the valuable and informative part inside the messages can be effectively extracted and leveraged to benefit the training of each individual agent. Unlike previous work, we do not introduce an extra decision module to dynamically adjust the communication pattern. This allows us to reduce the model complexity significantly. Instead, each agent first makes a preliminary decision based on its local information, and initiates communication only when its confidence level on this preliminary decision is low. Similarly, upon receiving the communication request, the agent replies to the request only when its message is informative. This architecture significantly improves the performance of the agents and reduces the communication overhead during the execution. Furthermore, it can be theoretically shown that the resulting training algorithm achieves guaranteed stability.

For evaluation, we test VBC on the StarCraft Multi-Agent Chanllenge~[15], the results demonstrate that VBC achieves $20\%$ higher for winning rates and $2-10\times$ lower for communication overhead on average compared with the other benchmark algorithms. A video demo is available at~[2] for better illustration of the VBC performance. The code is available at \url{https://github.com/saizhang0218/VBC}.

\section{Related Work}
The simplest training method for MARL is to make each agent learn independently using Independent Q-Learning (IQL)~[20]. Although IQL is successful in solving simple tasks such as Pong~[19], it ignores the environment dynamics arose from the interactions among the agents. As a result, it suffers from the problem of poor convergence, making it difficult to handle advanced tasks.

Given the recent success on deep Q-learning~[12], some work explores the scheme of centralized training and decentralized execution. Sunehag~\emph{et~al.}~[18] propose Value Decomposition Network (VDN), a method that acquires the joint action value function by summing all the action value functions of each agent. All the agents are trained as a whole by updating the joint action value functions iteratively. QMIX~[14] sheds some light on VDN, and utilizes a neural network to represent the joint action value function as a function of the individual action value functions and the global state information. The authors of~[10] extends the actor-critic methods to the multi-agent scenario. By performing centralized training and decentralized execution over the agents, the agents can better adapt to the changes on the environment and collaborate with each other. Foerster~\emph{et~al.}~[5] propose counterfactual multi-agent policy gradient (COMA), which employs a centralized critic function to estimate the action value function of the joint, and decentralized actor functions to make each agent execute independently. All the aforementioned methods assume no communication between the agents during the execution. As a result, many subsequent approaches, including ours, can be applied to improve the performance of these methods.

Learning the communication pattern for MARL is first proposed by Sukhbaatar~\textit{et.~al.}~[17]. The authors introduce \emph{CommNet}, a framework that adopts continuous communication for fully cooperative tasks. During the execution, each agent takes their internal states as well as the means of the internal states of the rest agents as the input to make decision on its action.
The BiCNet~[13] uses a bidirectional coordinated network to connect the agents. However, both schemes require all-to-all communication among the agents, which can cause a significant communication overhead and latency.

Several other proposals~[3,7,8] use a selection module to dynamically adjust the communication pattern between the agents. In~[3], the authors propose DIAL (Differentiable Inter-Agent Learning), the messages produced by an agent are selectively sent to the neighboring agents by the discretise/regularise unit (DRU). By jointly training DRU with the agent network, the communication overhead can be efficiently reduced. Jiang~\textit{et.~al.}~[7] propose an attentional communication model that learns when the communication is required and how to aggregate the shared information. However, an agent can only talk with the agents in its observable field at each timestep. This limits the speed of information propagation, and restricts the possible communication patterns when the local observable field is small. Kim~\textit{et.~al.} ~[8] propose a communication scheduling scheme for wireless environment, but only part of agents can broadcast their messages at each time.
In comparison, our approach does not apply the hard constraints on the communication pattern, which is advantageous to the learning process. Also our method does not adopt additional decision module for the communication scheduling, which greatly reduces the model complexity.

\section{Background}
\textbf{Deep Q-networks:}
We consider a standard reinforcement learning problem based on Markov Decision Process (MDP). At each timestamp $t$, the agent observes the state $s_{t}$, and chooses an action $a_{t}$. It then receives a reward $r_{t}$ for its action $a_{t}$ and proceeds to the next state $s_{t+1}$. The goal is to maximize the total expected discounted reward $R = \sum_{t=1}^{T} \gamma^{t} r_{t}$, where $\gamma\in [0,1]$ is the discount factor.
Deep Q-Networks (DQN) use a deep neural network to represent the action value function $Q_{\theta}(s,a) = E[R_{t}|s_{t} = s, a_{t} = a]$, where $\theta$ represents the parameters of the neural network, and $R_{t}$ is the total rewards received at and after $t$. During the training phase, a replay buffer is used to store the transition tuples $\big \langle s_{t},a_{t},s_{t+1},r_{t}\big \rangle$.
The action value function $Q_{\theta}(s,a)$ can be trained recursively by minimizing the loss $L = E_{s_{t},a_{t},r_{t},s_{t+1}}[y_{t}-Q_{\theta}(s_{t},a_{t})]^{2}$, where $y_{t} = r_{t} + \gamma max_{a_{t+1}} Q_{\theta'}(s_{t},a_{t+1})$ and $\theta'$ represents the parameters of the \emph{target network}.
An action is usually selected with $\epsilon$-greedy policy. Namely, selecting the action with maximum action value with probability $1-\epsilon$, and choosing a random action with probability $\epsilon$.

\textbf{Multi-agent deep reinforcement learning:}
We consider an environment with $N$ agents work cooperatively to fulfill a given task.
At timestep $t$, each agent $i$ ($1\leq i \leq N$) receives a local observation $o_{i}^{t}$ and executes an action $a_{i}^{t}$. They then receive a joint reward $r_{t}$ and proceed to the next state. We use a vector $\textbf{a}_{t}=\{a_{i}^{t}\}$ to represent the joint actions taken by all the agents. The agents aim to maximize the joint reward by choosing the best joint actions $\textbf{a}_{t}$ at each timestep $t$.

\textbf{Deep recurrent Q-networks:}
Traditional DQN generates action solely based on a limited number of local observations without considering the prior knowledge.
Hausknecht~\emph{et~al.}~[6] introduce Deep Recurrent Q-Networks (DRQN), which models the action value function with a recurrent neural network (RNN). The DRQN leverages its recurrent structure to integrate the previous observations and knowledge for better decision-making.
At each timestep $t$, the DRQN $Q_{\theta}(o^{t}_{i},h^{t-1}_{i},a^{t}_{i})$ takes the local observation $o^{t}_{i}$ and hidden state $h^{t-1}_{i}$ from the previous steps as input to yield action values.

\textbf{Learning the joint Q-function:}
Recent research effort has been made on the learning of joint action value function for multi-agent Q-learning. Two representative works are VDN~[14] and QMIX~[18]. In VDN, the joint action value function $Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t})$ is assumed to be the sum of all the individual action value functions $Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t}) = \sum_{i}Q_{i}(o_{i}^{t}, h_{i}^{t-1},a_{i}^{t})$, where $\textbf{o}_{t} = \{o_{i}^{t}\}$, $\textbf{h}_{t} = \{h_{i}^{t}\}$ and $\textbf{a}_{t} = \{a_{i}^{t}\}$ are the collection of the observations, hidden states and actions of all the agents at timestep $t$ respectively. QMIX employs a neural network to represent the joint value function $Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t})$ as a nonlinear function of $Q_{i}(o_{i}^{t}, h_{i}^{t-1},a_{i}^{t})$ and global state $s_{t}$.

\section{Variance Based Control}
In this section, we present the detailed design of VBC in the context of multi-agent Q-learning. The main idea of VBC is to achieve the superior performance and communication efficiency by limiting the variance of the transferred messages. Moreover,
during execution, each agent communicates with other agents only when its local decision is ambiguous. The degree of ambiguity is measured by the difference on top two largest action values. Upon receiving the communication request from other agents, the agent replies only if its feedback is informative, namely the variance of the feedback is high.
\subsection{Agent Network Design}
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{agent_network.pdf}
    \caption{(a) Agent network structure of agent 1, which consists of local agent generator, combiner and several message encoder. (b) The mixing network takes the output $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},a_{i}^{t})$ from each network of agent $i$, and perform centralized training. $c^{t}_{-i}$ means all the $c^{t}_{j\neq i}$.}
    \label{fig:network_system}
\end{figure}
The agent network consists of the following three networks: local action generator, message encoder and combiner. Figure~\ref{fig:network_system}(a) describes the network architecture for agent $1$. The \emph{local action generator} consists of a Gated Recurrent Unit (GRU) and a fully connected layer (FC). For network of agent $i$, the GRU takes the local observation $o_{i}^{t}$ and the hidden state $h_{i}^{t-1}$ as the inputs, and generates the intermediate results $c^{t}_{i}$. $c^{t}_{i}$ is then sent to the FC layer, which outputs the local action values $Q_{i}(o_{i}^{t},h_{i}^{t-1},a^{t}_{i})$ for each action $a^{t}_{i}\in A$, where $A$ is the set of possible actions.
The \emph{message encoder}, $f^{ij}_{enc}(.)$, is a multi-layer perceptrons (MLP) which contains two FC layers and a leaky ReLU layer. The agent network involves multiple independent message encoders, each accepts $c^{t}_{j}, j\neq i$ from another agent, and outputs $f_{enc}^{ij}(c^{t}_{j})$. The outputs from local action generator and message encoder are then sent to the \emph{combiner}, which produces the global action value function $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},a_{i}^{t})$  of agent $i$ by taking into account the global observation $\textbf{o}_{t}$ and global history $\textbf{h}_{t-1}$. To simplify the design and reduce model complexity, we do not introduce extra parameters for the combiner. Instead, we make the dimension of the $f^{ij}_{enc}(c^{t}_{j})$ the same as the local action values $Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$, and hence the combiner can simply perform elementwise summation over its inputs, namely $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},.) = Q_{i}({o}_{i}^{t},{h}_{i}^{t-1},.) + \sum_{j\neq i} f^{ij}_{enc}(c^{t}_{j})$. The combiner chooses the action with the $\epsilon$-greedy policy, $\pi(.)$.
Let $\theta_{local}^{i}$ and $\theta_{enc}^{ij}$ denote the set of parameters of the local action generators and the message encoders, respectively. To prevent the lazy agent problem~[18] and decrease the model complexity, we make $\theta_{enc}^{ij}$ is the same for all $i$ and $j(j\neq i)$, and also make $\theta_{local}^{i}$ the same for all $i$. Accordingly, we can drop the corner scripts and use $\boldsymbol\theta = \{\theta_{local},\theta_{enc}\}$ and $f_{enc}(.)$ to denote the agent network parameters and the message encoder.
\subsection{Loss Function Definition}
During the training phase, the message encoder and local action generator learn jointly to generate the best estimation on the action values. More specifically, we employ a mixing network (shown in Figure~\ref{fig:network_system}(b)) to aggregate the global action value functions $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},a_{i}^{t})$ from each agents $i$, and yields the joint action value function, $Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t})$.
To limit the variance of the messages from the other agents, we introduce an extra loss term on the variance of the outputs of the message encoders $f_{enc}(c^{t}_{j})$. The loss function during the training phase is defined as:
\begin{equation}
    \small
    L(\theta_{local},\theta_{enc}) = \sum_{b=1}^{B}\sum_{t=1}^{T}\big[(y^{b}_{tot}-Q_{tot}(\textbf{o}_{t}^{b},\textbf{h}_{t-1}^{b},\textbf{a}_{t}^{b};\boldsymbol\theta))^{2} + \lambda \sum_{i=1}^{N} Var(f_{enc}(c^{t,b}_{i}))\big]
    \label{eqn:loss}
\end{equation}
where $y^{b}_{tot} = r_{t}^{b} + \gamma max_{\textbf{a}_{t+1}}Q_{tot}(\textbf{o}_{t+1}^{b},\textbf{h}_{t}^{b},\textbf{a}_{t+1};\boldsymbol\theta^{-})$, $\boldsymbol\theta^{-}$ is the parameter of the target network which is copied from the $\boldsymbol\theta$ periodically, $Var(.)$ is the variance function and $\lambda$ is the weight of the loss on it. $b$ is the batch index.
The replay buffer is refreshed periodically by running each agent network and selecting the action which maximizes  $Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$.

\subsection{Communication Protocol Design}
During the execution phase, for every timestep $t$, the agent $i$ first computes the local action value function $Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$ and $f_{enc}(c_{i}^{t})$. It then measures the confidence level on the local decision by computing the difference between the largest and the second largest element within the action values. An example is given in Figure~\ref{fig:comm_protocol}(a). Assume agent 1 has three actions to select, and the output of the local action generator of agent $1$ is $Q_{1}(o_{1}^{t},h_{1}^{t-1},.) = (0.1,1.6,3.8)$, and the difference between the largest and the second largest action values is $3.8-1.6 = 2.2$, which is greater than the threshold $\delta_{1} = 1.0$. Given the fact that the variance of message encoder outputs $f_{enc}(c^{t}_{j})$ from the agent 2 and 3 is relatively small, due to the additional penalty term on variance in equation~\ref{eqn:loss}, it is highly possible that the global action value function $Q_{1}(\textbf{o}_{1}^{t},\textbf{h}_{1}^{t-1},.)$ also has the largest value in its third element. Therefore agent 1 does not have to talk to other agents to acquire $f_{enc}(c^{t}_{j})$. On the other hand, agent 1 broadcasts a request to ask for help if its confidence level on the local decision is low. Because the request does not contain any actual data, it consumes very low bandwidth. Upon receiving the request, only the agents whose message has a large variance reply (Figure~\ref{fig:comm_protocol}(b)), since their messages may change the current action decision at agent 1. This protocol not only reduces the communication overhead considerably, but also eliminates less informative messages which may impair the performance. Each agent only consults with the other agents when its confidence level on the local decision is low, and the other agents only reply when their messages can potentially change the final decision. The detailed protocol and operations performed at the agent unit is summarized in Algorithm~\ref{alg:communication_proto}.
\begin{figure}
    \centering
    \includegraphics[width=\columnwidth]{comm_example_new.pdf}
    \caption{An example on communication protocol of the agents during execution.}
    \label{fig:comm_protocol}
\end{figure}
\begin{algorithm}[tp!]
{
  \small
  \textbf{Input}: Threshold on confidence of local actions
  $\delta_{1}$, threshold on variance of message encoder output $\delta_{2}$. Total number of agents N.\\

    \For {$t\in T$}{
         $\backslash\backslash$ \textbf{Decision on the action of itself}: \\
         Compute local action values $Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$. Denote $m_{1},m_{2}$ the top two largest values of $Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$. \\
        \If {$m_{1}-m_{2}\geq \delta_{1}$}{
            Let $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},.)= Q_{i}(o_{i}^{t},h_{i}^{t-1},.)$.
        }
        \Else{
            Broadcast a request to the other agents, and receive the $f_{enc}(c_{j}^{t})$ from $N_{reply} (N_{reply}\leq N)$ agents. \\
            Let $Q_{i}(\textbf{o}_{t},\textbf{h}_{t-1},.)= Q_{i}(\textbf{o}_{i}^{t},\textbf{h}_{i}^{t-1},.) + \sum_{j=1}^{N_{reply}} f_{enc}(c_{j}^{t})$.
        }
        $\backslash\backslash$ \textbf{Generation of the exchanged message for the other agents}: \\
        Calculate variance of $f_{enc}(c_{i}^{t})$, if $Var(f_{enc}(c_{i}^{t}))\geq \delta_{2}$, store $f_{enc}(c_{i}^{t})$ in the buffer. \\
        \If {$Var(f_{enc}(c_{i}^{t}))\geq \delta_{2}$ \textbf{and} Receive a request from agent $j$}{
            {
               Reply the request from agent j with $f_{enc}(c_{i}^{t})$. \\
            }
        }

    }
	\caption{Communication protocol at agent $i$}
	\label{alg:communication_proto}
}
\end{algorithm}
\section{Convergence Analysis}
In this section, we analyze convergence of the learning process with the loss function defined in equation~(\ref{eqn:loss}) under the tabular setting. For the sake of simplicity, we ignore the dependency of the action value function on the previous knowledge $\textbf{h}_{t}$.
To minimize equation~(\ref{eqn:loss}), given the initial state $Q_{0}$, at iteration $k$, the $q$ values in the table is updated according to the following rule:
\begin{equation}
    \small
    Q_{tot}^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}\bigg[r_{t} + \gamma max_{\textbf{a}}Q_{tot}^{k}(\textbf{o}_{t+1},\textbf{a})-Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t}) - \lambda\sum_{i=1}^{N}\frac{\partial Var(f_{enc}(c^{t}_{i}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})}\bigg]
    \label{eqn:update_rule}
\end{equation}
where $\eta_{k}$, $Q_{tot}^{k}(.)$ are the learning rate and the joint action value function at iteration $k$ respectively, and $Q_{tot}^{*}(.)$ is the optimal joint action value function. We have the following result on the convergence of the learning process. A detailed proof is given in the appendix.
\begin{theorem}
    Assume $0\leq \eta_{k} \leq 1$, $\sum_{k}\eta_{k}=\infty$, $\sum_{k}\eta_{k}^{2}<\infty$. Also assume the number of possible actions and states are finite. By performing equation~\ref{eqn:update_rule} iteratively, we have $||Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})||\leq \lambda NG$ $\forall \textbf{o}_{t},\textbf{a}_{t}$, as $k \rightarrow \infty$, where $G$ satisfies $||\frac{\partial Var(f_{enc}(c^{t}_{i}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})}||\leq G, \forall i,k,t,\textbf{o}_{t},\textbf{a}_{t}$.
\end{theorem}

\begin{comment}
\begin{proof}
If we subtract $Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})$ from equation~\ref{eqn:update_rule}, and rearrange the equation, we get:
\begin{equation}
        \delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = (1-\eta_{k})\delta^{k}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}\bigg[r_{t} + \gamma max_{\textbf{a}}Q_{tot}^{k}(\textbf{o}_{t+1},\textbf{a})- Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t}) - \lambda\sum_{n=1}^{N}\frac{\partial Var(f_{enc}(c^{t}_{n}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})}\bigg]
\end{equation}
where $\delta^{k}(\textbf{o}_{t},\textbf{a}_{t}) = Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})$. Let $r_{t} + \gamma max_{\textbf{a}}Q_{tot}^{k}(\textbf{o}_{t+1},\textbf{a})- Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t}) = F_{k}(\textbf{o}_{t},\textbf{a}_{t})$, and $\sum_{n=1}^{N}\frac{\partial Var(f_{enc}(c^{t}_{n}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})} = U_{k}(\textbf{o}_{t},\textbf{a}_{t})$ we have:
\begin{equation}
        \delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = (1-\eta_{k})\delta^{t}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}\bigg[F_{k}(\textbf{o}_{t},\textbf{a}_{t}) - \lambda U_{k}(\textbf{o}_{t},\textbf{a}_{t})\bigg]
\end{equation}
Decompose $\delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t})$ into two random processes, $\delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t})$ and $\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$, where $\delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = \delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$, we have the following two random iterative processes:
\begin{align}
    \delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t}) &= (1-\eta_{k})\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}F_{k}(\textbf{o}_{t},\textbf{a}_{t}) \\
    \delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t}) &= (1-\eta_{k})\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t}) - \eta_{k}\lambda U_{k}(\textbf{o}_{t},\textbf{a}_{t})
\end{align}
From Theorem 1 of~\cite{melo2001convergence}, we know that $\delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t})$ converges to zero w.p. 1. From equation $6$, we notice that:
\begin{align}
     ||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| &\leq ||(1-\eta_{k})\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| + \eta_{k}\lambda ||U_{t}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq(1-\eta_{k})||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| + \eta_{k}\lambda NG
\end{align}
Therefore we have $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||-\lambda NG \leq(1-\eta_{k})(||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| -\lambda NG)$.
This is linear in $\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$ and $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||-\lambda NG$ will converge to a nonpositive number as k approaches infinity. However, because $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||$ is always greater or equal to zero, hence $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||$ must converge to a number between 0 and $\lambda NG$.
Therefore we get:
\begin{align}
     ||Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})|| &=||\delta^{k}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &=||\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq ||\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t})|| + ||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq \lambda NG
\end{align}
as k approaches infinity.
\end{proof}
\end{comment}

\section{Experiment}
We evaluated the performance of VBC with the StarCraft Multi-Agent Challenge (SMAC)~[15].
StarCraft II~[1] is a real-time strategy (RTS) game that has recently been utilized as a benchmark by the reinforcement learning community~[14,5,3,4]. In this work, we focus on the~\emph{decentralized micromanagement problem} in StarCraft II, which involves two armies, one controlled by the user (i.e. a group of agents), and the other controlled by the build-in StarCraft II AI.
The goal of the user is to control its allied units to destroy all enemy units, while minimizing received damage on each unit.
We consider six different battle settings. Three of them are \emph{symmetrical battles}, where both the user and the enemy groups consist of 2 Stalkers and 3 Zealots (2s3z), 3 Stalkers and 5 Zealots (2s5z), and 1 Medivac, 2 Marauders and 7 Marines (MMM) respectively. The other three are \emph{unsymmetrical battles}, where the user and enemy groups have different army unit compositions, including: 3 Stalkers for user versus 4 Zealots for enemy (3s$\_$vs$\_$4z), 6 Hydralisks for user versus 8 Zealots for enemy (6s$\_$vs$\_$8z), and 6 Zealot for user versus 24 Zerglings for enemy (6z$\_$vs$\_$24zerg). The unsymmetrical battles is considered to be harder than the symmetrical battles because of the difference in army size.

At each timestep, each agent controls a single unit to perform an action, including move[direction], attack[enemy$\_$id], stop and no-op. Each agent has a limited \emph{sight range} and \emph{shooting range}, where shooting range is less than the sight range. The attack operation is available only when the enemies are within the shooting range. The joint reward received by the allied units equals to the total damage inflicted on enemy units. Additionally, the agents are rewarded 100 extra points after killing each enemy unit, and 200 extra points for killing the entire army. The user group wins the battle only when the allied units kill all the enemies within the time limit. Otherwise the built-in AI wins.
The input observation of each agent is a vector consists the following information of each allied unit and enemy unit in its sight range: relative $x$, $y$ coordinates, relative distance and agent type. For the detailed game settings, hyperparameters, and additional experiment evaluation over other test environment, please refer to appendix.
\begin{comment}
During execution, each agent are allowed to perform the following actions: move[direction], attack[enemy$\_$id], stop and no-op. There are four directions for the 'move' operation: east, west, south, or north. Each agent can only attack the enemy within its \emph{shooting range}. The Medivacs can only heal the partner by performing the heal[partner$\_$id] instead of attacking the enemies. The number of possible actions for an agent ranges from 11 (2s3z) to 30 (6z$\_$vs$\_$24zerg). Each agent has a limited \emph{sight range}, and can only receive information from the partners or enemies within the sight range. Furthermore, the shooting range is smaller than the sight range so the agent can not attack the opponents without observing them, and the attack operation is not available when the opponents are outside the shooting range. Finally, agents can only observe the live agents within the sight range, and can not discern the agents that are dead or outside the range. At each timestep, the joint reward received by the allied units equals to the total damage on the health levels of the enemy units. Additionally, the agents are rewarded 100 extra points after killing each enemy unit, and 200 extra points for killing all the enemies. The user group win the battle only when the allied units kill all the enemies within the time limit. The user group lose the battle if either all the allied units are killed, or the time limit reaches. The time limit for different battles are: 120 timesteps for 2s3z and MMM, 150 for 3s5z and 3s$\_$vs$\_$4z, and 200 for 6h$\_$vs$\_$8z and 6z$\_$vs$\_$24zerg.

The input observation of each agent consisting of a vector which involves the following information of each allied unit and enemy unit in its sight range: relative x,y coordinates, relative distance and agent type. For the mixing network of QMIX, the global state vector $\textbf{s}$ contains the following elements:
\begin{enumerate}
    \item Shield levels, health levels and cooldown levels of all the units.
    \item The actions taken by all the units at the last timestep.
    \item The x,y coordinates of all the units relative to the center of the map.
\end{enumerate}
For all the six battles, each allied or enemy agent has a sight range of 9 and shooting range of 6 for all types of agents. For additional information, please refer to~\cite{samvelyan19smac}.
\end{comment}

\begin{comment}
\begin{table}
\small
\centering
\caption{Agent types of the six battles}
\begin{tabular}{c|c|c|c}

\textbf{Symm.} & \textbf{2s3z} & \textbf{MMM} & \textbf{3s5z}  \\ \hline
User  & 2 Stalkers \& 3 Zealots & 1 Medivac, 2 Marauders \& 7 Marines & 3 Stalkers \& 5 Zealots \\\hline
Enemy  & 2 Stalkers \& 3 Zealots & 1 Medivac, 2 Marauders \& 7 Marines & 3 Stalkers \& 5 Zealots \\\hline
\textbf{Unsymm.} & \textbf{3s\_vs\_4z} & \textbf{6h\_vs\_8z} & \textbf{6z\_vs\_24zerg} \\ \hline
User  & 3 Stalkers  &  6 Hydralisks & 6 Zealots \\\hline
Enemy  & 4 Zealots &   8 Zealots &  24 Zerglings
\end{tabular}
\label{table:game_setting}
\end{table}
\end{comment}

\subsection{Results}
We compare VBC and other banchmark algorithms, including
VDN~[18], QMIX~[14] and SchedNet~[8], for controlling allied units. We consider two types of VBCs by adopting the mixing networks of VDN and QMIX, denoted as VBC+VDN and VBC+QMIX. The mixing network of VDN simply computes the elementwise summation across all the inputs, and the mixing network of QMIX deploys a neural network whose weight is derived from the global state $\textbf{s}_{t}$. The detailed architecture of this mixing network can be found in~[14]. Additionally, we remove the penalty in Equation~(\ref{eqn:loss}), and drop the limit on variance during the execution phase (\emph{i.e.}, $\delta_{1} = \infty$ and $\delta_{2} = -\infty$). The agents are trained with the same network architecture shown in Figure~(\ref{fig:network_system}), and the mixing network of VDN is used. We call this algorithm FC (full communication). For SchedNet, at every timestep only $K$ out of $N$ agents can broadcast their messages by using $\emph{Top(k)}$ scheduling policy~[8]. $K$ changes with different battles and we usually set $K$ close to $0.5N$, that is, each time roughly half of the allied units can broadcast their messages. The VBC are trained for different number of episodes based on the difficulties of the battles, which we describe in detail next.

To measure the speed of convergence for each algorithm, we stop the training process and save the current model every 200 training episodes. We then run 20 test episodes and measure the winning rates for these 20 episodes. For VBC+VDN and VBC+QMIX, the winning rates are measured by running the communication protocol described in Algorithm~\ref{alg:communication_proto}. For easy tasks, namely MMM and 2s$\_$vs$\_$3z, we train the algorithms with $2$ million and $4$ million episodes respectively. For all the other tasks, we train the algorithms with $10$ million episodes. Each algorithm is trained 15 times. Figure~\ref{fig:winning_rates} shows the average winning rate and 95$\%$ confidence interval of each algorithm for all the six tasks.
For hyperparameters used by VBC (\emph{i.e.}, $\lambda$ used in equation~(\ref{eqn:loss}), $\delta_{1},\delta_{2}$ in Algorithm~\ref{alg:communication_proto}), we first search for a coarse parameter range based on random trial, experience and message statistics. We then perform a random search within a smaller hyperparameter space. Best selections are shown in the legend of each figure.

We notice that the algorithms that involve communication (\emph{i.e.}, SchedNet, FC, VBC) outperform the algorithms without communication (\emph{i.e.}, VDN, QMIX) in all the six tasks. This is a clear indication that communication benefits the performance. Moreover, both VBC+VDN and VBC+QMIX achieve better winning rates than SchedNet, because SchedNet only allows a fixed number of agents to talk at every timestep, which prohibits some key information to exchange timely. Finally, VBC achieves a similar performance as FC and even outplays FC for some tasks (\emph{e.g.}, 2s3z,6h$\_$vs$\_$8z, 6z$\_$vs$\_$24zerg). This is because a fair amount of communication between the agents are noisy and redundant. By eliminating these undesired messages, VBC is able to achieve both communication efficiency and performance gain.

\begin{figure*}[tp!]
    \centering
    \begin{subfigure}{.333\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{mmm_crop.pdf}
        \subcaption{MMM}\label{fig:MMM}
    \end{subfigure}%
    \begin{subfigure}{0.33\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{2s3z_crop.pdf}
        \subcaption{2s3z}\label{fig:2s3z}
    \end{subfigure}
    \begin{subfigure}{0.33\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{3s5z_crop.pdf}
        \subcaption{3s5z}\label{fig:3s5z}
    \end{subfigure}
    \begin{subfigure}{.333\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{3s_vs_4z_fff.pdf}
        \subcaption{3s\_vs\_4z}\label{fig:3s_vs_4z}
    \end{subfigure}%
    \begin{subfigure}{0.33\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{6h_vs_8z_fff.pdf}
        \subcaption{6h\_vs\_8z}\label{fig:6h_vs_8z}
    \end{subfigure}
    \begin{subfigure}{0.33\textwidth}
        \centering
        \includegraphics[width=0.92\linewidth, height=0.7\linewidth]{corridor_fff.pdf}
        \subcaption{6z\_vs\_24zerg}\label{fig:6z_vs_24zerg}
    \end{subfigure}
    \caption{Winning rates for the six tasks, the shaded regions represent the 95\% confidence intervals.}
    \label{fig:winning_rates}
\end{figure*}
\begin{figure*}[htb]
    \centering
    \begin{subfigure}{.3\textwidth}
        \centering
        \includegraphics[width=0.85\linewidth, height=0.6\linewidth]{comm_overhead.pdf}
        \caption{Communication overhead}
    \end{subfigure}%
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=0.78\linewidth, height=0.6\linewidth]{6hvs8zvbc_new.png}
        \caption{VBC (6h\_vs\_8z)}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=0.85\linewidth, height=0.6\linewidth]{6hvs8zqmix.png}
        \caption{QMIX and VDN (6h\_vs\_8z)}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=0.85\linewidth, height=0.65\linewidth]{ss5_new.png}
        \caption{VBC (3s$\_$vs$\_$4z)}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=0.8\linewidth, height=0.65\linewidth]{ss7_new.png}
        \caption{VBC (6z\_vs\_24zergs, stage 1)}
    \end{subfigure}
    \begin{subfigure}{0.3\textwidth}
        \centering
        \includegraphics[width=0.83\linewidth, height=0.65\linewidth]{ss8_new.png}
        \caption{VBC (6z\_vs\_24zergs, stage 2)}
    \end{subfigure}
    \caption{Strategies and communication pattern for different scenarios}
    \label{fig:strategies_and_comm}
\end{figure*}
\subsection{Communication Overhead}
We now evaluate the communication overhead of VBC. To quantify the amount of communication, we run Algorithm~\ref{alg:communication_proto} and count the total number of pairs of agents $g_{t}$ that conduct communication for each timestep $t$, then divided by the total number of pairs of agents in the user group, $R$. For example, for the task 3s$\_$vs$\_$4z, the user controls 3 Stalkers, and therefore the total number of possible pairs of agents $R$ for communication is $3\times 2 = 6$. Within these $6$ pairs of agents, suppose that $2$ pairs involve communication, and therefore $g_{t} = 2$. Let $\beta = \sum_{t} g_{t}/RT$ denote the communication overhead. Table~\ref{table:communication_overhead} shows the $\beta$ of VBC+VDN, VBC+QMIX and SchedNet across all the test episodes at the end of the training phase of each battle. For SchedNet, $\beta$ simply equals the ratio between number of allied agents that are allowed to talk and the total number of allied agents. As shown in Table~\ref{table:communication_overhead}, in contrast to ScheNet, VBC+VDN and VBC+QMIX produce 10$\times$ lower communication overhead for MMM and 2s3z, and $2-6\times$ less traffic for the rest of tasks.

\begin{figure}[bp!]
    \centering
    \begin{subfigure}{0.4\textwidth}
        \centering
        \includegraphics[width=0.9\linewidth, height=0.6\linewidth]{bar_fff.pdf}

    \end{subfigure}
    \begin{subfigure}{0.5\textwidth}
    \captionsetup{skip=0pt,font=scriptsize}
    \subcaption*{(b) Results on Cooperative Navigation (\#agents = 6)}
    \centering
    \begin{tabular}{c|c|c}
    \hline
    \enspace\enspace Methods  &  Avg. dist &  \#collisions \\\hline
    VBC+VDN  & \enspace 2.687 & \enspace\enspace 0.169 \\
    \enspace SchedNet  & \enspace 2.798 & \enspace\enspace 0.176 \\
    \enspace FC        & \enspace 2.990 & \enspace\enspace 0.161 \\
    \enspace\enspace VDN  & \enspace 3.886 & \enspace\enspace 1.872 \\
    \hline
    \end{tabular}
    \end{subfigure}
    \caption{(a) Results on PP with 3 predators and 3 prey. (b) shows results of CN.}
    \label{fig:maddpg-result2}
\end{figure}

\begin{table}
\centering
\caption{Communication overhead}
\begin{tabular}{|c | c | c | c|}
\hline
$\beta$ & VBC+VDN & VBC+QMIX & SchedNet \\ \hline
\textbf{MMM}  & 5.25\% & 5.36\% & 50\% \\
\textbf{2s3z}  & 4.33\% & 4.68\% & 60\% \\
\textbf{3s5z}  & 27.70\% & 28.13\% & 62.5\% \\
\textbf{3s\_vs\_4z}  & 5.07\% & 5.19\% & 33.3\% \\
\textbf{6h\_vs\_8z}  & 35.93\% & 36.16\% & 66.7\% \\
\textbf{6z\_vs\_24zerg}  & 12.13\% & 13.35\% & 50\% \\\hline
\end{tabular}
\label{table:communication_overhead}
\end{table}

\subsection{Learned Strategy}
\label{sec:strategy}
In this section, we examine the behaviors of the agents in order
to better understand the strategies adopted by the different algorithms. We have made a video demo available at~[2] for better illustration.

For unsymmetrical battles, the number of allied units is less than the enemy units,
and therefore the agents are prone to be besieged and killed by the enemies. This is exactly what happened for the QMIX and VDN agents on 6h$\_$vs$\_$8z, as shown in (Figure~\ref{fig:strategies_and_comm}(c)).
Figure~\ref{fig:strategies_and_comm}(b) shows the strategy of VBC, all the Hydralisks are placed in a row at the bottom margin of the map. Due to the limited size of the map, the Zealots can not go beyond the margin to surround the Hydralisks. The Hydralisks then focus their firings to kill each Zealot. Figure~\ref{fig:strategies_and_comm}(a) shows the change on $\beta$ for a sample test episode. We observe that most of the communication appears in the beginning of the episode. This is due to the fact that Hydralisks need to talk in order to place in a row. After the arrangement is formed, no communication is needed until the arrangement is broken due to the deaths of some Hydralisks, as indicated by the short spikes near the end of the episode.
Finally, SchedNet and FC utilize a similar strategy as VBC. Nonetheless, due to the restriction on communication pattern, the row formed by the allied agents are usually not well formed, and can be easily broken by the enemies.

For 3s$\_$vs$\_$4z scenario, the Stalkers have a larger attack range than Zealots.
All the algorithms adopt a kiting strategy where the Stalkers form a group and attack the Zealots while kiting them. For VBC and FC, at each timestep only the agents that are far from the enemies attack, and rest of the agents (usually the healthier ones) are used as a shield to protect the firing agents (Figure~\ref{fig:strategies_and_comm}(d)). Communication only occurs when the group are broken and need to realign. In contrast, VDN and QMIX do not have this attacking pattern, and all the Stalkers always fire simultaneously, therefore the Stalkers closest to the Zealots are get killed soon. SchedNet and FC also adopt a similar policy as VBC, but the attacking pattern of the Stalkers is less regular, i.e., the Stalkers close to the Zealots also fire occasionally.

6z$\_$vs$\_$24zerg is the toughest scenario in our experiment. For QMIX and VDN, the 6 Zealots are surrounded and killed by 24 Zerglings shortly after episode starts. In contrast, VBC first separates the agents into two groups with two Zealots and four Zealots respectively (Figure~\ref{fig:strategies_and_comm}(e)). The two Zealots attract most of Zerglings to a place far away from the rest four Zealots, and are killed shortly. Due to the limit sight range of the Zerglings, they can not find the rest four Zealots. On the other side, the four Zealots kill the small part of Zerglings easily and search for the rest Zerglings. The four Zealots take advantage of the short sight of the Zerglings.
Each time the four Zealots adjust their positions in a way such that they can only be seen by a small amount of the Zerglings, the baited Zerglings are then be killed easily (Figure~\ref{fig:strategies_and_comm}(f)). For VBC, the communication only occurs in the beginning of the episode when the Zealots are separated into two groups, and near the end of the episode when four Zealots adjust their positions. Both FC and SchedNet learn the strategy of splitting the Zealots into two groups, but they fail to fine-tune their positions to kill the remaining Zerglings.

For symmetrical battles, the tasks are less challenging, and we see less disparities on performances of the algorithms. For 2s3z and 3s5z, the VDN agents attack the enemies blindly without any cooperation. The QMIX agents learn to focus firing and protect the Stalkers. The agents of VBC, FC and SchedNet adopt a more aggressive policy, where the allied Zealots try to surround and kill the enemy Zealots first, and then attack the enemy Stalkers by collaborating with the allied Stalkers. This is extremely effective because Zealots counter Stalkers, so it is important to kill the enemy Zealots before they damage allied Stalkers. For VBC, the communication occurs mostly when the allied Zealots try to surround the enemy Zealots. For MMM, almost all the methods learn the optimal policy, namely killing the Medivac first, then attack the rest of the enemy units cooperatively.
\subsection{Evaluation on Cooperative Navigation and Predator-prey}

Furthermore, we evaluate the algorithms for two more scenarios: (1) Cooperative Navigation (CN) which is a cooperative scenario, and (2) Predator-prey (PP) which is a competitive scenario. The game settings are the same as what are used in~[10] and~[8], respectively. We train each method until convergence and test the result models for 2000 episodes. For PP, we make the agents of VBC to compete against the agents of other methods, and report the normalized score of Predator (Figure~\ref{fig:maddpg-result2}(a)). For CN we report the average distance between agents and their destinations, and average number of collisions (Figure~\ref{fig:maddpg-result2}(b)). We notice that methods which allows communication (\emph{i.e.}, SchedNet, FC, VBC) outperform the others for both tasks, and VBC achieves the best performance. Moreover, VBC incurs a communication overhead of $10.07\%$ and $8.80\%$ for PP and CN respectively. In CN, most of the communication of VBC occurs when the agents are close to each other to prevent collisions. In PP, the communication of VBC occurs mainly to rearrange agent positions for better coordination.

\section{Conclusion}
In this work, we propose VBC, a simple and effective approach to achieve efficient communication among the agents in MARL. By constraining the variance of the exchanged messages during the training phase, VBC improves communication efficiency while enabling better cooperation among the agents.
The test results of StarCraft Multi-Agent Challenge indicate that VBC outperforms the other state-of-the-art methods significantly in terms of both winning rate and communication overhead.

\newpage
\section{Appendix}

\subsection{Proof for theorem 1}
\begin{theorem}
    Assume $0\leq \eta_{k} \leq 1$, $\sum_{k}\eta_{k}=\infty$, $\sum_{k}\eta_{k}^{2}<\infty$. Also assume the number of possible actions and states are finite. By performing equation 2 iteratively, we have $||Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})||\leq \lambda NG$ $\forall \textbf{o}_{t},\textbf{a}_{t}$, as $k \rightarrow \infty$, where $G$ satisfies $||\frac{\partial Var(f_{enc}(c^{t}_{i}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})}||\leq G, \forall i,k,t,\textbf{o}_{t},\textbf{a}_{t}$.
\end{theorem}
\begin{proof}
The proof is based on~[21] and ~[22].
If we subtract $Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})$ from equation 2 in the paper, and rearrange the equation, we get:
\begin{equation}
        \delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = (1-\eta_{k})\delta^{k}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}\bigg[r_{t} + \gamma max_{\textbf{a}}Q_{tot}^{k}(\textbf{o}_{t+1},\textbf{a})
         - Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t}) - \lambda\sum_{i=1}^{N}\frac{\partial Var(f_{enc}(c^{t}_{i}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})}\bigg]
\end{equation}
where $\delta^{k}(\textbf{o}_{t},\textbf{a}_{t}) = Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})$. Let $r_{t} + \gamma max_{\textbf{a}}Q_{tot}^{k}(\textbf{o}_{t+1},\textbf{a})- Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t}) = F_{k}(\textbf{o}_{t},\textbf{a}_{t})$, and $\sum_{i=1}^{N}\frac{\partial Var(f_{enc}(c^{t}_{i}))}{\partial Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})} = U_{k}(\textbf{o}_{t},\textbf{a}_{t})$ we have:
\begin{equation}
        \delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = (1-\eta_{k})\delta^{t}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}\bigg[F_{k}(\textbf{o}_{t},\textbf{a}_{t}) - \lambda U_{k}(\textbf{o}_{t},\textbf{a}_{t})\bigg]
\end{equation}
Decompose $\delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t})$ into two random processes, $\delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t})$ and $\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$, where $\delta^{k+1}(\textbf{o}_{t},\textbf{a}_{t}) = \delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$, we have the following two random iterative processes:
\begin{align}
    \delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t}) &= (1-\eta_{k})\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \eta_{k}F_{k}(\textbf{o}_{t},\textbf{a}_{t}) \\
    \delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t}) &= (1-\eta_{k})\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t}) - \eta_{k}\lambda U_{k}(\textbf{o}_{t},\textbf{a}_{t})
\end{align}
From Theorem 1 of~[22], we know that $\delta^{k+1}_{1}(\textbf{o}_{t},\textbf{a}_{t})$ converges to zero w.p. 1. From equation $6$, we notice that:
\begin{align}
     ||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| &\leq ||(1-\eta_{k})\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| + \eta_{k}\lambda ||U_{t}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq(1-\eta_{k})||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| + \eta_{k}\lambda NG
\end{align}
Therefore we have $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||-\lambda NG \leq(1-\eta_{k})(||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| -\lambda NG)$.
This is linear in $\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})$ and $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||-\lambda NG$ will converge to a nonpositive number as k approaches infinity. However, because $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||$ is always greater or equal to zero, hence $||\delta^{k+1}_{2}(\textbf{o}_{t},\textbf{a}_{t})||$ must converge to a number between 0 and $\lambda NG$.
Therefore we get:
\begin{align}
     & ||Q_{tot}^{k}(\textbf{o}_{t},\textbf{a}_{t})-Q_{tot}^{*}(\textbf{o}_{t},\textbf{a}_{t})|| =||\delta^{k}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &=||\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t}) + \delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq ||\delta^{k}_{1}(\textbf{o}_{t},\textbf{a}_{t})|| + ||\delta^{k}_{2}(\textbf{o}_{t},\textbf{a}_{t})|| \\
     &\leq \lambda NG
\end{align}
as k approaches infinity.
\end{proof}

\begin{comment}
\section{VBC training algorithm}
The detailed training algorithm for VBC is shown in Algorithm 2.
\begin{algorithm}[tph]
{

	\textbf{Input}: Initial parameters $\{\theta_{local},\theta_{enc}\}$ for the agent network, and the mixing network. The initial parameters $\theta_{mix}$ for the mixing network. The discount factor $\gamma$, size of minibatch B. \\
	\For {training episode $e \in E$}{
	     Observe the initial state $s_{0}$. \\
	     \For {timestep $t \in T$} {
	         Run Algorithm 1 for each agent and get the action value vector $Q_{i}(\textbf{o}^{t},\textbf{h}^{t-1},.)$ for each agent i. \\
	         Pick the actions $\textbf{a}^{t}= \{a_{i}^{t}\}$ which maximize $Q_{i}(\textbf{o}^{t},\textbf{h}^{t-1},.)$ for each agent i.
	         Execute the actions and observes the joint reward $r_{t}$ and next state $s_{t+1}$. \\
	         Store the tuple $(s_{t}, \textbf{a}^{t}, r_{t}, s_{t+1})$ to the replay buffer. \\
	     }
	     Sample a minibatch S from the replay buffer. \\
	     For each sample within the minibatch, get the observation $o_{t}^{i}$ for each agent i, using the agent network to compute $max_{a_{i}^{t}}Q_{i}(\textbf{o}^{t},\textbf{h}^{t-1},.)$. Then use the action values as the input to mixing network, which then yields the joint action value function  $Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t})$ at t. \\
	     Compute $y = r_{t} + \gamma  Q_{tot}(\textbf{o}_{t},\textbf{h}_{t-1},\textbf{a}_{t})$, and $y'= Q_{tot}(\textbf{o}_{t+1},\textbf{h}_{t},\textbf{a}_{t+1})$. \\
	     Calculate the loss $L = \sum_{b=1}^{B} ||y'-y||^{2}$. Update $\theta_{local},\theta_{enc}, \theta_{mix}$.
    }
}
\label{alg:training_alg}
\caption{Training algorithm of VBC}
\end{algorithm}
\end{comment}

\section{Experiment settings and hyperparameters}
In this section, we describe in detail the experiment settings.
\subsection{StarCraft micromanagement challenges}
StarCraft II~[1] is a real-time strategy (RTS) game that has recently been utilized as a challenging benchmark by the reinforcement learning community~[14,5,13]. In this work, we concentrate on the~\emph{decentralized micromanagement problem} in StarCraft II. Specifically, the user controls an army that consists of several army units (agents). We consider the battle scenario where two armies, one controlled by the user, and the other controlled by the build-in StarCraft II AI, are placed on the same map and try to defeat each other. The agent type can be different between the two armies, and the agent type can also be different within the same army. The goal of the user is to control the allied units wisely to kill all the enemy units, while minimizing the damage on the health of each individual agent. The difficulty of the computer AI is set to \emph{Medium}. We consider six different battle settings, which is shown in Table~\ref{table:game_setting}. Among these six settings, three are \emph{symmetrical battles}, where the user group and the enemy group are identical in terms of type and quantity of agents. The other three are \emph{unsymmetrical battles}, where the agent type of user groups and enemy group are different, and user group contains less number of allied units than the enemy group. The unsymmetrical battles is consider to be harder than the symmetrical battles because of the difference in the army size.

During execution, each agent are allowed to perform the following actions: move[direction], attack[enemy$\_$id], stop and no-op. There are four directions for the 'move' operation: east, west, south, or north. Each agent can only attack the enemy within its \emph{shooting range}. The Medivacs can only heal the partner by performing the heal[partner$\_$id] instead of attacking the enemies. The number of possible actions for an agent ranges from 11 (2s3z) to 30 (6z$\_$vs$\_$24zerg). Each agent has a limited \emph{sight range}, and can only receive information from the partners or enemies within the sight range. Furthermore, the shooting range is smaller than the sight range so the agent can not attack the opponents without observing them, and the attack operation is not available when the opponents are outside the shooting range. Finally, agents can only observe the live agents within the sight range, and can not discern the agents that are dead or outside the range. At each timestep, the joint reward received by the allied units equals to the total damage on the health levels of the enemy units. Additionally, the agents are rewarded 100 extra points after killing each enemy unit, and 200 extra points for killing all the enemies. The user group wins the battle only when the allied units kill all the enemies within the time limit. The user group loses the battle if either all the allied units are killed, or the time limit reaches. The time limit for different battles are: 120 timesteps for 2s3z and MMM, 150 for 3s5z and 3s$\_$vs$\_$4z, and 200 for 6h$\_$vs$\_$8z and 6z$\_$vs$\_$24zerg.

The input observation of each agent consisting of a vector which involves the following information of each allied unit and enemy unit in its sight range: relative x,y coordinates, relative distance and agent type. For the mixing network of QMIX, the global state vector $\textbf{s}_{t}$ contains the following elements:
\begin{enumerate}
    \item Shield levels, health levels and cooldown levels of all the units at $t$.
    \item The actions taken by all the units at $t-1$.
    \item The x,y coordinates of all the units relative to the center of the map at $t$.
\end{enumerate}
For all the six battles, each allied or enemy agent has a sight range of 9 and shooting range of 6 for all types of agents. For additional information, please refer to~[15].
\begin{table*}
\small
\centering
\caption{Agent types of the six battles}
\begin{tabular}{c|c|c|c}

\textbf{Symm.} & \textbf{2s3z} & \textbf{MMM} & \textbf{3s5z}  \\ \hline
User  & 2 Stalkers \& 3 Zealots & 1 Medivac, 2 Marauders \& 7 Marines & 3 Stalkers \& 5 Zealots \\\hline
Enemy  & 2 Stalkers \& 3 Zealots & 1 Medivac, 2 Marauders \& 7 Marines & 3 Stalkers \& 5 Zealots \\\hline
\textbf{Unsymm.} & \textbf{3s\_vs\_4z} & \textbf{6h\_vs\_8z} & \textbf{6z\_vs\_24zerg} \\ \hline
User  & 3 Stalkers  &  6 Hydralisks & 6 Zealots \\\hline
Enemy  & 4 Zealots &   8 Zealots &  24 Zerglings
\end{tabular}
\label{table:game_setting}
\end{table*}
\subsection{Hyperparameter}
For network of agent $i$, at timestep $t$, raw observation $o_{i}^{t}$ is first passed through a single-layer MLP, which outputs a intermediate result with size of 64. The GRU then takes this intermediate result, as well as the hidden states $h^{t-1}_{i}$ from the previous timestep and generates $h^{t}_{i}$ and $c_{i}^{t}$. Both $h^{t}_{i}$ and $c_{i}^{t}$ has a size of 64.  The $c_{i}^{t}$ is then passed through a FC layer, which generates the local action value function $Q_{i}(o_{i}^{t},h_{i}^{t},.)$. The message encoder contains two FC layers with 196 and 64 units respectively. The combiner performs elementwise summation on the outputs of local action generator and the message encoders.

During the training, we set the $\gamma = 0.99$ and decrease $\epsilon$ linearly from 1.0 to 0.05 over the first $200000$ timesteps and keep it to $0.05$ for the rest of the learning process. The replay buffers stores the most recent 5000 episode. We perform a test run for every 200 training episodes to update the replay buffer. The training batch size is set to 32 and the test batch size is set to 8. We adopt a RMSprop optimizer with a learning rate $\eta=5\times 10^{-4}$ and $\alpha = 0.99$.

\section*{Reference}
{[1]}\enspace Starcraft official game site: \url{https://starcraft2.com/}.

{[2]}\enspace VBC Video demo: \url{https://bit.ly/2VFkvCZ}.

{[3]}\enspace J.~Foerster, I.~A. Assael, N.~de~Freitas, and S.~Whiteson.
\newblock Learning to communicate with deep multi-agent reinforcement learning.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  2137--2145, 2016.

{[4]}\enspace J.~N. Foerster, C.~A.~S. de~Witt, G.~Farquhar, P.~H. Torr, W.~Boehmer, and
  S.~Whiteson.
\newblock Multi-agent common knowledge reinforcement learning.
\newblock {\em arXiv preprint arXiv:1810.11702}, 2018.

{[5]}\enspace J.~N. Foerster, G.~Farquhar, T.~Afouras, N.~Nardelli, and S.~Whiteson.
\newblock Counterfactual multi-agent policy gradients.
\newblock In {\em Thirty-Second AAAI Conference on Artificial Intelligence},
  2018.

{[6]}\enspace M.~Hausknecht and P.~Stone.
\newblock Deep recurrent q-learning for partially observable mdps.
\newblock In {\em 2015 AAAI Fall Symposium Series}, 2015.

{[7]}\enspace J.~Jiang and Z.~Lu.
\newblock Learning attentional communication for multi-agent cooperation.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  7254--7264, 2018.

{[8]}\enspace D.~Kim, S.~Moon, D.~Hostallero, W.~J. Kang, T.~Lee, K.~Son, and Y.~Yi.
\newblock Learning to schedule communication in multi-agent reinforcement
  learning.
\newblock {\em arXiv preprint arXiv:1902.01554}, 2019.

{[9]}\enspace J.~Kober, J.~A. Bagnell, and J.~Peters.
\newblock "reinforcement learning in robotics: A survey.".
\newblock {\em The International Journal of Robotics Research}, 2013.

{[10]}\enspace R.~Lowe, Y.~Wu, A.~Tamar, J.~Harb, O.~P. Abbeel, and I.~Mordatch.
\newblock Multi-agent actor-critic for mixed cooperative-competitive
  environments.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  6379--6390, 2017.

{[11]}\enspace L.~Marc, V.~Zambaldi, A.~Gruslys, A.~Lazaridou, K.~Tuyls, J.~Pérolat,
  D.~Silver, and T.~Graepel.
\newblock "a unified game-theoretic approach to multiagent reinforcement
  learning.".
\newblock In {\em Advances in Neural Information Processing Systems}, 2017.

{[12]}\enspace V.~Mnih, K.~Kavukcuoglu, D.~Silver, A.~Graves, I.~Antonoglou, D.~Wierstra, and
  M.~Riedmiller.
\newblock "playing atari with deep reinforcement learning.".
\newblock {\em arXiv preprint arXiv:1312.5602}, 2013.

{[13]}\enspace P.~Peng, Y.~Wen, Y.~Yang, Q.~Yuan, Z.~Tang, H.~Long, and J.~Wang.
\newblock Multiagent bidirectionally-coordinated nets: Emergence of human-level
  coordination in learning to play starcraft combat games.
\newblock {\em arXiv preprint arXiv:1703.10069}, 2017.

{[14]}\enspace T.~Rashid, M.~Samvelyan, C.~S. de~Witt, G.~Farquhar, J.~Foerster, and
  S.~Whiteson.
\newblock Qmix: Monotonic value function factorisation for deep multi-agent
  reinforcement learning.
\newblock {\em arXiv preprint arXiv:1803.11485}, 2018.

{[15]}\enspace M.~Samvelyan, T.~Rashid, C.~S. de~Witt, G.~Farquhar, N.~Nardelli, T.~G.~J.
  Rudner, C.-M. Hung, P.~H.~S. Torr, J.~Foerster, and S.~Whiteson.
\newblock {The} {StarCraft} {Multi}-{Agent} {Challenge}.
\newblock {\em CoRR}, abs/1902.04043, 2019.

{[16]}\enspace S.-S. Shai, S.~Shammah, and A.~Shashua.
\newblock "safe, multi-agent, reinforcement learning for autonomous driving.".
\newblock {\em arXiv preprint arXiv:1610.03295}, 2016.

{[17]}\enspace S.~Sukhbaatar, R.~Fergus, et~al.
\newblock Learning multiagent communication with backpropagation.
\newblock In {\em Advances in Neural Information Processing Systems}, pages
  2244--2252, 2016.

{[18]}\enspace P.~Sunehag, G.~Lever, A.~Gruslys, W.~M. Czarnecki, V.~Zambaldi, M.~Jaderberg,
  M.~Lanctot, N.~Sonnerat, J.~Z. Leibo, K.~Tuyls, et~al.
\newblock Value-decomposition networks for cooperative multi-agent learning.
\newblock {\em arXiv preprint arXiv:1706.05296}, 2017.

{[19]}\enspace A.~Tampuu, T.~Matiisen, D.~Kodelja, I.~Kuzovkin, K.~Korjus, J.~Aru, J.~Aru, and
  R.~Vicente.
\newblock Multiagent cooperation and competition with deep reinforcement
  learning.
\newblock {\em PloS one}, 12(4):e0172395, 2017.

{[20]}\enspace M.~Tan.
\newblock "multi-agent reinforcement learning: Independent vs. cooperative
  agents.".
\newblock In {\em Proceedings of the tenth international conference on machine
  learning.} IEEE, 1993.

{[21]}\enspace T.~Jaakkola, M.~I. Jordan, and S.~P. Singh.
\newblock Convergence of stochastic iterative dynamic programming algorithms.
\newblock In {\em Advances in neural information processing systems}, pages
  703--710, 1994.

{[22]}\enspace F.~S. Melo.
\newblock Convergence of q-learning: A simple proof.
\newblock {\em Institute Of Systems and Robotics, Tech. Rep}, pages 1--4, 2001.

\end{document}

redpajama_set_name: RedPajamaArXiv


While walk-in closets have the most applicable features, reach-in closets have the most organizational needs. After all, if you are trying to fit an entire wardrobe into a smaller space. Bottom line, you need an efficient way to do it. Here at More Space Place Jacksonville, that's our passion. Allow us to help you make your reach-in feel like a walk-in with a variety of luxurious features to maximize your space.
Having a small space means that every inch makes a big difference. If your current closet arrangement is not working for you, don't settle! Instead, you can focus on the organization features that solve your storage problems. Do you extra hanging space for your summer clothes? We can install double rods to get you just that. Do you need deep drawers for bulky jeans and sweaters? You've got it! We can also install accessory drawers and racks, custom shoe storage, and more. Just let us know what pieces of your wardrobe you need easily accessible and we'll handle the rest.
Generally, the most underused parts of the closet sit at the ceiling and the floor. If you can build up these spaces in a fashion that suits you, there will be more room and better organization. Here at More Space Place Jacksonville, we want to make sure to use every inch of space in our custom closets. Consider high shelves to store seasonal items that you don't use regularly. On the floor, we can install individual cubbies or a long rack for organizing shoes. Avoid the mountain of shoes with one of these easy to use systems.
Most closets come with a hanging rod. Does yours work effectively? You may be able to improve it. If you have a lot of shirts, trousers or skirts that you need to hang, you could easily double your hanging space by adding a second rod. Some people like to install one at eye level and one halfway down. Others prefer the concept of the pull-down rod, to take an item off the rack and replace it quickly. Children's closets might even support a tiered system of rods.
The whole point of a good closet is to stay organized and make everything quick to find. Easier to reach inside is an important element of a reach-in closet. Pull-out concepts simplify the repetitive task of getting things in the morning or at night. Drawers that tilt or pull-out help you maximize a deep closet. Pull-out racks or shelving make grabbing what you need a snap, without disrupting anything else nearby. Lastly, a pull-out laundry hamper can make sure you don't confuse clean for dirty and keep items separate.
Closet organization that meets your needs could be a game changer for your home. By adding the right components in an efficient design, you can dramatically increase your storage potential, while making your whole day easier. Get started today with a free in-home design consultation to measure and plan your space with one of our associates. We proudly serve the greater Jacksonville area. Contact us today!


text:
Virgin Australia increases domestic flying as new safety and wellbeing measures are introduced
The Virgin Australia Group will increase domestic flying and introduce new safety and wellbeing measures as state travel restrictions begin to ease and more travellers return to the skies.
The additional services will see Virgin Australia double current capacity by early July, adding approximately 30,000 seats across 320 flights per week to its domestic schedule.
• Expanded capital city connections and frequencies between Sydney, Brisbane, Melbourne and Perth
• Resumption of services between Brisbane-Adelaide, Brisbane-Canberra, Brisbane-Perth and Sydney-Perth
• More flexibility to travel with increased frequencies on intrastate Queensland and Western Australia routes
• New set of safety and wellbeing measures to minimize risks associated with COVID-19
Virgin Australia will also introduce a comprehensive new set of safety and wellbeing measures including pre-departure eligibility and health screening, contactless check-in, more frequent cleaning onboard and at the airport, and expanded social distancing measures.
The measures, which will be fully implemented by 12 June have been developed in close collaboration with airlines and airports across Australia as part of the Australian Aviation Recovery Coalition. The measures will continue to be reviewed in line with any changes to its schedule, the latest medical advice, government restrictions and guest feedback.
Virgin Australia Group Chief Commercial Officer John MacLeod said as state travel restrictions further ease, demand for air travel has started to return which is why more services and increased frequencies are being added.
"By early July, we will have gradually added approximately 30,000 seats across 320 flights per week to our schedule – more than doubling our capacity and providing more flexibility for guests.
"It's early days but these services will be a welcomed boost to Australia's tourism industry and help the nation's economy and aviation sector to rebuild. The services will also give us the opportunity to bring team members back to work, who I know have been looking forward to welcoming and sharing our award-winning customer service.
"Importantly as travel begins to increase, the safety and wellbeing of guests remains crucial, which is why measures to minimise risks associated with COVID-19 are being introduced. We recommend that any guest who is feeling unwell or has flu-like symptoms, do not travel – flights for travel up until 30 September can be changed without fees if needed."
Virgin Australia Group Medical Officer Dr Sara Souter said the new measures will ensure that we appropriately manage the latest advice from government and remain vigilant when it comes to hygiene and personal protection.
"We will be adopting contactless check-in, a new sequenced boarding and disembarkation process and guests will see more frequent cleaning of high touch surfaces on the aircraft and within the airport. In addition, a new health questionnaire will be rolled out as part of the check-in process to ensure passengers are fit to fly and to assist with contact tracing," said Dr Souter.
"It's important to remember that the inflight environment is very different to that of other modes of group transport, with a range of protective features including hospital-grade air filtration systems, directional airflow and high seat backs between travellers. The international evidence supports very low-risk of COVID-19 transmission inflight, which is also our experience to-date in Australia.
"Everyone has a role to play, which is why we are asking guests to be mindful of their own personal protection and others when moving around the cabin. Team members and guests are being encouraged to regularly wash their hands and avoid touching their face, and hand sanitiser and masks will be available to all guests on request.
"Wherever possible, we will try to do our best to keep an empty seat between guests travelling alone, however this may not always be available. Families and travelling companions will be able to sit together."
Guests travelling are encouraged to download the Australian Government's COVIDSafe tracing app. Guests should also ensure their details are updated via the My Bookings function at virginaustralia.com or with their travel agent prior to commencing travel.
Virgin Australia aircraft photo gallery:
This entry was posted in Virgin Australia Airlines and tagged Virgin Australia Airlines on June 11, 2020 by Bruce Drum.
← Spirit AeroSystems furloughs 900 employees due to Boeing order to slow MAX production Air Transat to resume operations on July 23 →

redpajama_set_name: RedPajamaCommonCrawl


text:
Annual report and accounts Overview
Imperial in 2018–19
Risks and risk management approach
Principles of remuneration
Responsibilities of the Council
Council and Council Committees
Financial statements 2018–19
Annual report and accounts archive
Annual report and accounts 2017-18
Imperial in 2017-18
The dual leadership model
Select Provostial contributions
Imperial's first Provost, Professor James Stirling, and Professor Julia Higgins with Dr Jessica Wade and Dr Cleo Kontoravdi - winners of the 2017 Julia Higgins Award, which recognises their contribution to the support of academic women at the College.
A strong senior leadership team
The Provost has recruited and led a high performing senior academic leadership team. This has ensured the sustained excellence of Imperial in its core academic mission of research, education and innovation.
Innovative teaching for world class learning
The Provost led the development and approval of Imperial's Learning and Teaching Strategy, with £8 millon of funding to enable improvement of Imperial's education curriculum, student experience, and online education provision. This contributed to the award last year of an institutional Gold in the first Teaching Excellence Framework.
The Provost has renewed focus on multi-disciplinary research, with a review of Imperial's cross-faculty Institutes, Networks and Centres, and a new process for coordinating large cross-College research bids.
Supportive student services
The Provost has overhauled student support services, including the appointment of the College's first Director of Student Services, who has overall responsibility for the provision of Student Services.
Improved financial planning
The Provost has provided strong academic oversight and leadership in close partnership with the Chief Financial Officer for the annual financial planning round. Meetings with each Dean, Vice-Provost and support service budget-holder align the budget with the core academic mission of College, and the strategic vision of the President.
Promoting gender equality
The Provost has been a tireless champion of gender equality at Imperial, and in STEM more generally, since arriving at the College. This credible, senior academic leadership for gender equality at Imperial has been pivotal, along with a revised governance structure led by the Provost.
All departments are now required to be recognised, or applying to be recognised, by the Athena SWAN charter, with 18 of our 21 departments now achieving at least Bronze accreditation. The Provost led the successful renewal of our institutional Silver award in 2017.
New approach to animal research
The Provost led the transformation of animal research at Imperial, creating the College's 2014 Action Plan for Animal Research. The change in culture, renewed spirit of collaboration between academic and technical staff, revised governance structure, cultivation of productive working relationships with the Home Office (our regulator), and investment in facilities have made the College a nationally recognised exemplar in quality animal research.
Valuing all staff
The Provost championed the development and wellbeing of a number of cohorts of Imperial's staff community, including post-doctoral researchers and technicians.
Making mental health and wellbeing a priority
Following the 2014 Staff Survey, the Provost led a series of initiatives creating a step-change in the promotion of mental health and wellbeing of our staff. The College now has a network of over 300 trained Mental Health First Aiders in addition to 45 Staff Supporters, to highlight and signpost the support offered by the College for those suffering mental ill-health.
The College also runs a series of events during National Mental Health Awareness Week.
Download the Annual Report and Accounts [PDF, 20MB]
Download the five-year summary of key statistics [PDF, 44KB]


text:
Swimming News and Highlights
ELIZABETH MARKS EARNS SILVER IN THE 50-METER FREESTYLE
Aug. 25, 2021 - Winning medals is nothing new for Elizabeth Marks. After all, the three-time Paralympian has a couple to her name. But earning them for a stroke that is not in her wheelhouse is something foreign to her.
Enlisted Airman Competes for Team USA in 2019 CISM Military World Games
Dec. 3, 2019 - He was the youngest and only enlisted member from a team of 14 service members from all branches of the Department of Defense on the U.S. swim team to compete at the 2019 CISM Military World Games.
Wasil earns gold, two bronze in CISM Swimming/Para-Swimming Open
Feb. 18, 2015 - Fontainebleau, France – Competing in the men's division, Army Sgt. Elizabeth Wasil captures gold in the Para-Swimming 100m breaststroke with a time of 1:27.66 at the 2015 Conseil International du Sport Militaire (CISM) World Military Swimming & Para-Swimming Open at the French National Defense Sports Center Feb. 4-5.


Construction of a much-awaited seaport in the coastal province of Kampot began yesterday, with insiders saying the upcoming harbour will boost maritime transportation with Vietnam and Thailand and bolster the local tourism industry and the national economy.
The facility is being built with a concessional loan from the Asian Development Bank, which was disbursed to improve connectivity with neighbouring countries in southern provinces, according to Tourism Minister Thong Khon, who presided over the groundbreaking ceremony yesterday.
The Governor of Kampot province, Cheav Tay, said the port will prove particularly useful for tourists wishing to connect to destinations in Vietnam and Thailand.
"The development of the port will not only contribute to the development of the local infrastructure; it will benefit the country's economy as well.
"This port will be a very important facility, connecting countries in the region. We do expect to see an increase in tourism in the coast once it is in operation," he said.
Soy Sinol, director of Kampot's tourism department, told Khmer Times in a previous interview that the port will be completed by the end of 2019.
In 2017, Cambodia received more than 5.6 million international tourists, an increase of 11.8 percent, earning more than $3.6 billion in revenue (nearly 12 percent of GDP), and directly employing over 650,000 people.
That year, the coast alone welcomed 3.2 million tourists, with international visitors numbered at 740,000, a hike of 20 percent.


What Role Should Threat Intelligence Play?
In order to respond to the latest threats you need to know what those threats are.
One of the biggest areas of growth in the security industry in the last couple of years has been around threat intelligence. While many organisations will buy or subscribe to threat intelligence information, the real challenge is knowing how to integrate it into your people and processes.
In this video Chris Yule, SecureWorks Senior Principal Consultant, gives a quick overview into the importance of threat intelligence and the challenges of effective integration.
In order to respond to the latest threats you need to know what those threats are. One of the biggest areas of growth in the security industry in the last couple of years has been around threat intelligence. We're seeing organizations who've bought into threat intelligence that are struggling to know what to do with it. They've either bought product that they think will protect them or they've bought a heap of threat intelligence subscriptions that they don't know what to do with. We're working with a lot of organizations to help them build that capability into their people and processes where they can take that threat intelligence, analyze it to figure out what's really important to them and tailor the output of that to their business so they can really adapt their security and invest in all of the right areas to protect themselves properly.


text:
Enemy è un film del 2013 diretto da Denis Villeneuve e con protagonista Jake Gyllenhaal. La pellicola è liberamente tratta dal romanzo L'uomo duplicato (2002) di José Saramago.

Trama
Un uomo assiste a uno spettacolo erotico in un club sotterraneo, che culmina con una donna nuda in procinto di schiacciare una tarantola.

Adam Bell, un impacciato professore universitario di storia, vive una vita tranquilla e monotona in una città dominata da un cielo costantemente grigio. Su consiglio di un collega, noleggia un film di produzione locale, Volere è potere, notandovi con grande sorpresa una comparsa che sembra essere il suo sosia perfetto. Cercando online, identifica l'attore come Daniel St. Claire, nome d'arte di Anthony Claire. Adam affitta gli altri due film in cui ha recitato Anthony e ne diventa ossessionato, trascurando il lavoro e la sua ragazza, Mary. In casa sua ritrova anche una foto di qualcuno che gli assomiglia, con una mano femminile sulla sua spalla: tuttavia, parte della foto è strappata, rendendo impossibile identificare la donna. Adam ha anche dei frequenti sogni in cui compare una tarantola.

Recatosi all'agenzia di Anthony, viene scambiato per quest'ultimo e ritira una busta da cui ne ricava l'indirizzo. Lo chiama a casa, ma le sue farneticazioni finiscono solo per spaventare la moglie incinta di Anthony, Helen. Quest'ultima, convinta che si tratti dell'ennesima infedeltà del marito, pedina Adam, rimanendo profondamente turbata dalla loro somiglianza. Alla fine, Anthony accetta di incontrarsi con lui in una camera d'albergo, dove constatano la rispettiva identicità, fin nella medesima cicatrice sulla pancia. Quando Anthony lo incalza riguardo a una possibile parentela segreta, Adam è preso alla sprovvista dalla personalità opposta dell'altro e, messo a disagio, se ne va. Il giorno dopo, è Anthony che comincia a seguire Adam fino ad adocchiare Mary. Invaghitosi di lei, Anthony accusa Adam di aver avuto rapporti sessuali con Helen, sostenendo che lui in cambio dovrebbe permettergli di fare altrettanto con Mary.

Si fa consegnare quindi i suoi vestiti e le chiavi della sua macchina per poter trascorrere una serata con lei, promettendo poi di sparire per sempre dalla vita di Adam. Mentre Anthony porta Mary nella camera d'albergo dell'incontro, Adam si reca nell'appartamento dell'altro, dove viene scambiato per Anthony dal portiere, che gli chiede disperatamente di riportarlo nel club sotterraneo. Nell'appartamento, Adam trova una foto incorniciata su una mensola, simile a quella di casa sua, ma intatta e con Helen assieme a lui. Adam cerca di fingersi Anthony di fronte a Helen, ma, anche se non sembra notare le sottili differenze o il comportamento timido, questa gli chiede comunque come è andata la giornata a scuola. Più tardi quella notte, Helen consola Adam, che le chiede scusa in lacrime: lei gli chiede di rimanere e i due fanno l'amore. Nel frattempo, Mary e Anthony stanno avendo un rapporto sessuale, quando questa si accorge del segno lasciato sulla pelle dalla fede nuziale di Anthony e va nel panico, realizzando la verità. Costringe Anthony a riportarla a casa in macchina, ma durante il tragitto i due litigano furiosamente, finendo per fare un incidente, presumibilmente mortale.

Il giorno dopo, Adam si veste con gli abiti di Anthony e apre la busta ritirata in precedenza dall'agenzia, trovandovi la chiave del club sotterraneo. Rientrato in camera da letto da Helen, scopre al suo posto una tarantola grande come la stanza e rannicchiata contro il muro posteriore.

Significato dell'opera
Film criptico in alcuni frangenti e pervaso di momenti non esattamente lineari, spesso insensati nelle scelte dei protagonisti finalizzati a rappresentare una critica verso le imposizioni sociali, verso gli "obblighi" morali che condizionano le scelte di un individuo e spingono l'evoluzione della sua esistenza in una direzione piuttosto che in un'altra. Da qui deriva il titolo del film: l'opera esprime infatti la concezione secondo cui ogni individuo sia l'unico vero nemico di se stesso.

Anthony e Adam sono rappresentati come due persone perfettamente identiche fisicamente ma completamente agli antipodi sotto ogni altro aspetto proprio per questo: Adam vive una relazione abbastanza libera con una fidanzata e ha un lavoro molto stabile, ma ha dovuto affrontare il fallimento di un matrimonio e ha messo da parte il proposito di diventare attore per assicurarsi una carriera più solida, come scopriamo dalla conversazione con sua madre; Anthony invece è un attore professionista e ha una moglie incinta prossima al parto, ma ha interpretato pochi ruoli marginali lavorativamente e nel privato ha un costante bisogno di evadere dalla vita coniugale trascorrendo del tempo in un club erotico. Queste scelte hanno generato un temperamento dimesso e timido in Adam e uno decisamente più focoso e aggressivo in Anthony, ma il risultato rimane simile: entrambi hanno rinunciato a una parte di sé per un'imposizione sociale, sia essa il matrimonio monogamo o la carriera assicurata.

Il ruolo delle donne del film è ancora più calato in quest'ottica, come se le inquadrasse come dei simboli delle convenzioni sociali: la madre che ha pretese e atteggiamenti piccolo-borghesi nei confronti del figlio, la moglie gelosa e che desidera che il suo uomo sia più simile ad Adam che ad Anthony, la fidanzata che concorre nel determinare un epilogo non felice per uno dei due. Il finale ambiguo del film sottolinea infine uno dei messaggi cardine dell'opera: un vero cambiamento dell'individuo è possibile solo qualora cambi effettivamente qualcosa dentro di lui.

Produzione

Le riprese del film sono cominciate il 12 maggio 2012 a Toronto.

Distribuzione
Il film è stato presentato in anteprima al Toronto International Film Festival l'8 settembre 2013. È stato distribuito nelle sale cinematografiche canadesi da Entertainment One a partire dal 14 marzo 2014 e in quelle spagnole da Alfa Pictures a partire dal 28 marzo dello stesso anno.

In Italia, il film è stato presentato al Noir in Festival il 13 dicembre 2013, dove ha vinto anche il Premio Nero come miglior film e distribuito direttamente in home video da Eagle Pictures il 24 ottobre 2017.

Accoglienza

Accoglienza
Secondo Rotten Tomatoes, il film ha ottenuto un indice di gradimento del 71% e un voto di 6,60 su 10 sulla base di 121 recensioni. Secondo Metacritic, il film ha ottenuto un voto di 61 su 100 sulla base di 30 recensioni.

Incassi
Il film ha incassato circa 3,4 milioni di dollari al botteghino.

Riconoscimenti
 2013 - Courmayeur Noir in festival
 Leone nero al miglior film
 2014 - Canadian Screen Awards
 Miglior regista a Denis Villeneuve
 Miglior attrice non protagonista a Sarah Gadon
 Miglior fotografia a Nicolas Bolduc
 Miglior montaggio a Matthew Hannam
 Miglior colonna sonora a Danny Bensi e Saunder Jurriaans
 Candidatura per il miglior film
 Candidatura per il miglior attore a Jake Gyllenhaal
 Candidatura per la miglior sceneggiatura non originale a Javier Gullón
 Candidatura per la miglior scenografia a Patrice Vermette
 Candidatura per i migliori effetti speciali
 2014 - Toronto Film Critics Association Awards
 Miglior film canadese

Note

Altri progetti

Collegamenti esterni

Film thriller psicologici
Film thriller erotici
Film basati su opere di narrativa
Adattamenti di opere di José Saramago
Film girati in Canada
Film ambientati a Toronto
Film indipendenti
Film diretti da Denis Villeneuve

redpajama_set_name: RedPajamaWikipedia


text:
Home Local Teens guilty of forging currency
Teens guilty of forging currency
Carol Winker
Curfew imposed until sentencing
A Grand Court
jury last week found three teen-aged boys guilty of forging Cayman
Islands currency notes, mostly of the $25 denomination. The notes
were produced on home computer equipment, including scanner and printer.
All three had pleaded not guilty to
the forgery charge.
However, one of them admitted possession.
After an incident at school that led police to question him, his mother found
24 $25 notes inside a teddy bear in the boy's bedroom. It was actually her
teddy bear and when she picked it up she saw it had been cut.
A fourth boy, charged with possession
and uttering counterfeit currency – passing it into circulation – was found not
guilty.
Because the boys were 13, 14 and 15
years old at the time of the offences, their names will not be reported. They
are referred to as A, B, C and D.
After the jury's verdicts, Justice
Roy Jones told the boys found guilty that the court would request social
inquiry reports on them, so they should return for sentencing on 19 February. Meanwhile,
they are to observe a curfew from 7pm to 6am.
Defence Attorney Nicholas Dixey
acknowledged that if it were adults before the court, they would have to be
advised of the possibility of a custodial sentence. But in this case, involving
young persons, he asked if their bail could not continue as before. Ordering a
curfew now seemed to be a form of punishment before sentence, he suggested.
The judge said that during the
trial bail had been based on the presumption of innocence. But now, "They have
been convicted. That means something."
The forged notes came to light in
September 2007.
In presenting the case for the
Prosecution, Crown Counsel John Masters first called Stephanie Scott. She told
the court she was assisting cashiers at the John Gray High School canteen.
On Friday, 7 September, she
received a $50 note. At the end of the day she noticed it was a fake. She
showed it to her father, who has the contract for the canteen service. They
decided to see if someone would hand over another fake note on Monday.
Another cashier had taken in a
counterfeit $25 on the Friday.
Monday, 10 September, was a typical
day with a lot of quick transactions.
Every cashier knew to feel the texture of the bill handed in, Ms Scott said.
Then she herself received a $25
note and felt it. She told the student who had given it to her that it was
fake. She said he replied, "It's not me, it's not me. Someone gave me the
She said she hung on to his book bag
and tried to get the attention of a security guard. By that time, the boy
"shrugged me off and took off – he basically ran out of the cafeteria."
Ms Scott reported the matter to the
deputy principal, Mr. Samaroo. A book of student photos was used to identify
the boy as A.
A was found and taken to the
office. He said he had passed noted on Friday and Monday on behalf of B and did
not know they were counterfeit.
B was then called to the office and
he denied involvement. Mr. Samaroo phoned the boys' parents and requested them
to come to the school. He also phoned police and handed over the two notes from
Friday and the one from Monday.
The officers spoke to the boys in
the presence of their parents and officers of the Family Support
Unit.
When B was asked where he got the
notes, his reply was "I nah getting um from nobody."
After his mother found the counterfeit
notes in his room, B was interviewed again. He denied being involved in the
producing of the notes, but admitted giving two notes to A.
Asked where he got them, B
implicated C and D.
Meanwhile a Bodden Town
businessman reported receiving a counterfeit $25.
Officers checking the home of C and
D found the genuine original note that the businessman had a fake copy of. They
also found the originals of the copies that had been handed in at the canteen.
Further, on examining a stack of
computer printing paper, one officer found a sheet on which a back and a front
of a $25 note had been printed. A paper cutter was also on the premises.
During interviews with police in
the presence of their parents, C and D said they had nothing to do with the
production of the counterfeit notes.
They said B came to their house several times. They suggested he could have
used the computer to print the notes while they were watching television in the
same room.
C's father spoke during the
interview and said he had seen B at the computer.
The boy A gave evidence during the trial. The
other three did not, relying on their interviews.
A agreed he passed two counterfeit
notes into circulation but said he did not know at the time that they were not
real. He denied running from the canteen, but said he went to find B to ask him
where the notes came from.
During addresses to the jury,
Defence Attorney Adam King pointed out that the canteen cashier had accepted
the fake note on the Friday. "The cashier was fooled, so you might think [A]
was too," Mr. King suggested.
The jury agreed and their not
guilty verdict was unanimous.
The guilty verdicts were also
unanimous.
Mr. Dixey, along with Mr. King and Defence
Attorney John Furniss, had queried why it took a year and a half before police
charged the boys.
Sergeant Michael Montaque said the
officer originally in charge of the case had left the force. After he took
over, there were several leads that had to be investigated. Then the information
had to be compiled and sent to the Legal Department.
In his summing up, Mr. Masters said
it's okay to criticise police when they get things wrong, but the public should
acknowledge when they get things right. In this case, police could have stopped
with A and B, but they continued to follow leads. In this case, they had done
not just a pretty good investigation but a superior investigation.
Bodden Town
Previous articleThe Big and Little of being a Mentor
Next articleGet gear fit for training
Historic Nixon house to be relocated and preserved
CUC gives update on Bodden Town power outages
Pole fire causes widespread power outages
CARIFESTA acts highlighted at Harquail
How the US Congress could destroy social media
Leadership Cayman applications sought


SIR ALEX FERGUSON reckons Wayne Rooney is back to his best after a summer of uncertainty.
Chelsea and Arsenal both wanted to buy Rooney in the summer but Manchester United refused to sell.
The striker was understood to want out of Old Trafford after his relationship with Ferguson deteriorated last season.
Rooney is out of contract in 2015 and Fergie, who is now a director at Old Trafford, has lavished praise on the man he was rumoured to have fallen out with.
Ferguson said: "On Tuesday night [against Bayer Leverkusen] they were terrific.
"Wayne's performance was fantastic, I was pleased to see that.
"He's got his energy back, the determination, his purpose to attack players - it was all very encouraging to me.
"Being a director I'm delighted to see that, he's back to what we always remember."


Are you about to develop a new system or product, or refine an existing one? Or does your customer require you to verify the security of your products/systems?
Cyber security isn't just important for IT infrastructures. Embedded systems are also increasingly exposed to threats of hacker attacks. The complexity of systems is rising, thus increasing the number of potential weak points.
New kinds of systems are often not protected by security templates, requiring a completely new security concept to be created. Internet-capable vehicles, for instance, do not use standard IT components, but vehicle-specific buses and operating systems.
The security requirements on manufacturer products are also becoming increasingly specific: automotive suppliers and manufacturers of energy systems, drive technology or medical technology, for instance, have to test the security of their products/systems before they are allowed to use them.
We offer security and penetration tests to identify these security issues and analyse the security mechanisms used. We systematically conduct tests and specifically search for potential points of attack. If any weak points are found, they will be assessed to determine the risk they pose and recommendations will be made as to how to eliminate them.
Your advantage: If you know the weak points in your systems, you can effectively protect yourself from internal and external attacks for the long term!
In the future, it will become increasingly important that information security is already taken into consideration when researching and developing new products, and that the principle of Security by Design is adhered to. CYOSS stands for security from the beginning of development to the delivery of the product: We develop, integrate and improve security mechanisms in all steps of the development process in close cooperation with our clients. The aim is to identify and prevent potential weaknesses already in the planning phase.
Your advantage: Security concepts during the system's development ensure the systematic identification of all assets, threats and possible measures.
Long-term experience with embedded systems, including in the automotive and aerospace industries – we know the perspective of the developers as well as that of the security experts.
We are masters in "security" as well as "safety" and can appraise the implications one has on the other.
For instance, we have developed the security concepts and requirements for the OEMs in the automotive industry which the suppliers have to put into practice and refine. That's how we know precisely how we can support you.