We consider scenarios from the real-time strategy game StarCraft as benchmarks for reinforcement learning algorithms. We focus on micromanagement, that is, the short-term, low-level control of team members during a battle. We propose several scenarios that are challenging for reinforcement learning algorithms because the stateaction space is very large, and there is no obvious feature representation for the value functions. We describe our approach to tackle the micromanagement scenarios with deep neural network controllers from raw state features given by the game engine. We also present a heuristic reinforcement learning algorithm which combines direct exploration in the policy space and backpropagation. This algorithm collects traces for learning using deterministic policies, which appears much more efficient than, e.g., -greedy exploration. Experiments show that this algorithm allows to successfully learn non-trivial strategies for scenarios with armies of up to 15 agents, where both Q-learning and REINFORCE struggle.
1 INTRODUCTION
StarCraft1 is a real-time strategy (RTS) game in which each player must build an army and control individual units to destroy the opponent’s army. As of today, StarCraft is considered one of the most difficult games for computers, and the best bots only reach the level of high amateur human players (Churchill, 2015). The main difficulty comes from the need to control a large number of units in partially observable environment, with very large state and action spaces: for example, in a typical game, there are at least 101685 possible states whereas the game of Go has about 10170 states. Because of simultaneous and durative actions, StarCraft provides an ideal environment to study the control of many agents at large scale, and an opportunity to define tasks of increasing difficulty, from micromanagement, which concerns the short-term, low-level control of fighting units during battles, to long-term strategic and hierarchical planning under uncertainty. While building a controller for the full game based on machine learning is out-of-reach with current methods, we propose, as a first step, to study reinforcement learning (RL) algorithms in micromanagement scenarios in StarCraft.
Both the work on Atari games (Mnih et al., 2013) and the recent Minecraft scenarios studied by researchers (Abel et al., 2016; Oh et al., 2016) focus on the control of a single agent, with a fixed, limited set of actions. Coherently controlling multiple agents (units) is the main challenge of reinforcement learning for micromanagement tasks. This comes with two main challenges. The first one is to efficiently explore the large action space. The implementation of a coherent strategy requires the units to take actions that depend on each other, but it also implies that any small alteration of a strategy must be maintained for a sufficiently long time to properly evaluate the long-term effect of that change. In contrast to this requirement of consistency in exploration, the reinforcement learning algorithms that have been successful in training deep neural network policies such as Q-learning (Watkins & Dayan, 1992; Sutton & Barto, 1998) and REINFORCE (Williams, 1992; Deisenroth et al., 2013), perform exploration by randomizing actions. In the case of micromanagement, randomizing actions mainly disorganizes the units, which then rapidly lose the battle without collecting relevant
*: These authors contributed equally to this work. 1StarCraft and its expansion StarCraft: Brood War are trademarks of Blizzard EntertainmentTM
feedback. The second challenge of micromanagement is that there is no obvious way to parameterize the policy given the state and the actions, because actions are relations between entities of the state, e.g. (unit A, attack, unit B) or (unit A, move, position B) and are not restricted to a few constant symbols such as “move left” or “move right”. Multi-class architectures, such as these used for Atari games (Mnih et al., 2015), cannot evaluate actions that are parameterized by an entity of the state.
The contribution of this paper is twofold. First, we propose several micromanagement tasks from StarCraft (Section 3), then we describe our approach to tackle them and evaluate well known reinforcement learning algorithms on these tasks (Section 4). In particular, we present an approach of greedy inference to break out the complexity of taking the actions at each step. We also describe the features used to jointly represent states and actions, as well as a deep neural network model for the policy (Section 5). Second, we propose the zero order (ZO) reinforcement learning algorithm to address the difficulty of exploration in these tasks (Section 6). Compared to algorithms for efficient direct exploration in parameter space, the novelty of our algorithm is to explore directly in policy space by mixing parameter randomization and plain gradient descent.
2 RELATED WORK
Multi-agent reinforcement learning has been an active area of research (Busoniu et al., 2008). Most of the focus has been on learning agents in competitive environments with adaptive adversaries (Littman, 1994; Hu & Wellman, 1998; Tesauro, 2003). Some work has looked at learning control policies for individual agents in a collaborative setting with communication constraints (Tan, 1993; Bernstein et al., 2002), with applications such as soccer robot control (Stone & Veloso, 1999), and methods such as hierarchical reinforcement learning for communicating high-level goals (Ghavamzadeh et al., 2006), or learning an efficient communication protocol (Sukhbaatar et al., 2016). While the decentralized control framework is most likely relevant for playing full games of StarCraft, here we avoid the difficulty of imperfect information, therefore we use the multi-agent structure only as a means to structure the action space. As in the approach of (Maes et al., 2009) with reinforcement learning for structured output prediction, we use a greedy sequential inference scheme at each time frame: each unit decides on its action based solely on the state combined with the actions of units that came before it in the sequence.
Algorithms that have been used to train deep neural network controllers in reinforcement learning include Q-learning (Watkins & Dayan, 1992; Mnih et al., 2015), the method of temporal differences (Sutton, 1988; Tesauro, 1995), policy gradient and their variants (Williams, 1992; Deisenroth et al., 2013), and actor/critic architectures (Barto et al., 1983; Silver et al., 2014; 2016). Except for the deterministic policy gradient (DPG) (Silver et al., 2014), these algorithms rely on randomizing the actions at each step for exploration. DPG collects traces by following deterministic policies that remain constant throughout an episode, but can only be applied when the action space is continuous. Hausknecht & Stone (2015) apply DPG with paramterized action spaces, in which discrete actions (e.g. “move”) are parameterized by continuous variables (e.g. the target location). Our work is most closely related to works that explore the parameter space of policies rather than the action space. Several approaches have been proposed that randomize the parameters of the policy at the beginning of an episode and run a deterministic policy throughout the entire episode, borrowing ideas from gradient-free optimization, e.g. (Mannor et al., 2003; Sehnke et al., 2008; Szita & Lörincz, 2006). However, these algorithms rely on gradient-free optimization for all parameters, which does not scale well with the number of parameters. Osband et al. (2016b) describe another type of algorithm where the parameters of a deterministic policy are randomized at the beginning of an episode, and learn a posterior distribution over the parameters as in Thomson sampling (Thompson, 1933). Their approach was proved to be efficient, but applies only to linear functions and scales quadratically with the number of parameters. The bootstrapped deep Q-networks (BDQN) (Osband et al., 2016a) are a practical implementation of the ideas of (Osband et al., 2016b) for deep neural networks. However, BDQN still performs exploration in the action space at the beginning of the training, and there is no randomization of the parameters. BDQN keeps several versions of the last layer of the deep neural network, and selects a single version per episode to perform Q-learning updates, while it ensembles all such “heads” as test time. In contrast, we randomize the parameters of the last layer once at the beginning of each episode and do not rely of estimates of a state-action value function.
In the context of RTS micromanagement, a large spectrum of AI approaches have been studied. There has been work on Bayesian fusion of hand-designed influence maps (Synnaeve & Bessiere, 2011), fast heuristic search in a simplified simulator (Churchill et al., 2012), and even evolutionary optimization (Liu et al., 2014). Overmind (Klein et al., 2010) used threat-aware A* pathing and RL-tuned potential fields. Closer to this work, Marthi et al. (2005) employ concurrent hierarchical Q-learning (units Q-functions are combined at the group level), Wender & Watson (2012) successfully applied tabular Q-learning (Watkins & Dayan, 1992) and SARSA (Sutton & Barto, 1998), with and without experience replay (“eligibility traces”), with a reward similar to the one used in several of our experiments. However, the action space was reduced to pre-computed “meta-actions”: fight and retreat, and the features were hand-crafted. None of these approaches are used as is in existing StarCraft bots, for a lack of robustness, completeness (both can be attributed to hand-crafting), or computational efficiency. For a more detailed overview of AI research on StarCraft, the reader should consult (Ontanón et al., 2013).
3 STARCRAFT MICROMANAGEMENT SCENARIOS
We focus on micromanagement, which consists of optimizing each unit’s actions during a battle. The tasks presented in this paper represent only a subset of the complexity of playing StarCraft. As StarCraft is a real-time strategy (RTS) game, actions are durative (are not fully executed on the next frame), and there are approximately 24 frames per second. As we take an action for each unit every few frames (e.g. every 9 frames here, more details can be found in Appendix D), we only consider actions that can be executed in this time frame, which are: the 8 move directions, holding the current position, an attack action for each of the existing enemy units. During training, we always control all units from one side, and the opponent (built-in AI in the experiments) is attacking us:
• m5v5 is a task in which we control 5 Marines (ranged ground unit), against 5 opponent Marines. A good strategy here is to focus fire, e.g. order all Marines to attack a single opponent.
• m15v16: same as above, except we have 15 Marines and the opponent has 16. A good strategy here is also to focus fire, while avoiding “overkill.” 7 Marines attacking simultaneously kills an opponent in a single volley, so using more marines to simultaneously target an enemy causes attacks to be wasted, resulting in “overkill.”
• dragoons_zealots: symmetric armies with two types of units: 3 Zealots (melee ground unit) and 2 Dragoons (ranged ground unit). Here a strategy requires to focus fire, and if possible to 1) not spend too much time having the Zealots walk instead of fight, 2) focus the Dragoons, who die more easily but deal more damage.
• w15v17: we control 15 Wraiths (ranged flying unit) while the opponent has 17. Flying units have no “collision”, so multiple units can occupy the same tile and reach their target more quickly. It only takes 6 wraiths to kill an opponent in a single volley. Hence, it is important not to “overkill” on this map.
• other mXvY or wXvY scenarios. The 4 scenarios above are the ones on which we train our models, but they can learn strategies that overfit a given number of units, so we have similar scenarios but with different numbers of units (on each side).
For all these scenarios, a human expert can win 100% of the time against the built-in AI, by moving away units that are hurt (thus conserving firepower) and with proper focus firing.
4 FRAMEWORK: RL AND MULTIPLE UNITS
Formalism The environment is approximated as a Markov Decision process (MDP), with a finite set of states denoted by S . Each state s has a set of units U(s), and a policy has to issue a command c ∈ C to each of them. The set of commands is finite. An action in that MDP is represented as a sequence of (unit, command) pairs a = ((u1, c1), ..., (u|s|, c|s|)) such that {u1, ..., u|s|} = U(s). |s| denotes the number of units in state s andA(s) = (U(s)×C)|s| the set of actions in state s. We denote by ρ(s′|s, a) the transition probability of the MDP and by ρ1 the probability distribution of initial states. When there is a transition from state st to a state st+1, the agent receives the reward rt+1 = r(st, st+1), where r : S × S → R is the reward function. We assume that commands are received and
executed concurrently, so that the order of commands in an action does not alter the transition probabilities. Finally, we consider the episodic reinforcement learning scenario, with finite horizon T and undiscounted rewards. The learner has to learn a (stochastic) policy π(a|s), which defines a probability distribution over actions in A(s) for every s ∈ S. The objective is to maximize the expected undiscounted cumulative reward over episodes R(π) = E[ ∑T−1 t=1 r(s
t, st+1)] = E[r̄1..T ], where the expectation is taken with respect to s1 ∼ ρ1, st+1 ∼ ρ(.|at, st) and at ∼ π(.|st).
The “greedy” MDP One way to break out the complexity of jointly inferring the commands to each individual unit is to perform greedy inference at each step: at each state, units choose a command one by one, knowing the commands that were previously taken by other units. Learning a greedy policy boils down to learning a policy in another MDP with fewer actions per state but exponentially more states, where the additional states correspond to the intermediate steps of the greedy inference. This reduction was previously proposed in the context of structured prediction by Maes et al. (2009), who proved that an optimal policy in this new MDP has the same cumulative reward as an optimal policy in the original MDP. We expand on this in Appendix B.
Normalized cumulative rewards Immediate rewards are necessary to provide feedback that guides exploration. In the case of micromanagement, a natural reward signal is the difference between damage inflicted and incurred between two states. The cumulative reward over an episode is the total damage inflicted minus the total damage incurred along the episode. However, the scale of this quantity heavily depends on the number of units (both our units and enemy units, which significantly decreases along an episode) that are present in the state. Without proper normalization with respect to the number of units in the current state z(s), learning will be artificially biased towards the large immediate rewards at the beginning of the episode. Then, instead of considering cumulative rewards from a starting state st, we define normalized cumulative rewards n̄t..T as the following recursive computation over an episode:
∀t ∈ {1, ..., T − 1}, n̄t..T = r t+1 + z(st+1)n̄t+1..T
z(st) . (1)
We use the sum of maximum hit points of all units in the state st as normalization factor z(st), which implies that n̄t..T ∈ [−0.5, 0.5]. One way to look at this normalization process is to consider that the reward is r t+1
z(st) , and z(st+1) z(st) plays the role of an (adaptive) discount factor, which is chosen to be at
most 1, and strictly smaller than 1 when the number of units change.
For policy gradient and our algorithm described in section 6, we directly use n̄t..T . We describe in Appendix C how we adapted the update rule for Q-learning.
5 FEATURES AND MODEL FOR MICROMANAGEMENT IN STARCRAFT
We describe in this section the features and the neural network architecture we use to parameterize the policy. Since we consider the greedy inference described in the previous section, the underlying MDP will contain states of the form s̃ = (s, a1..k, uk+1), where: s is the current state of the game given by the game engine, k is the number of units which already “played” at this frame, a1..k is the sequence of the k pairs (unit, command) that correspond to the k commands the have already been chosen, and finally uk+1 is the unit to play. For each unit, we consider two types of commands: (1) attack a given enemy unit, and (2) move to a specific position. In order to reduce the number of possible move commands, we only consider 9 move commands, which either correspond to a move in one of the 8 basic directions, or staying at the same position.
There are several challenges to represent states and actions in RTS games:
• The number of units and actions are not bound a priori and varies in time • Commands must be evaluated in context of all currently executing commands • Attack actions must resolve the reference to its target
To address the first two challenges, we adopt an approach based on a joint encoding of states and commands. Denoting by s̃ = (s, a1..k, uk+1) the current state of the greedy MDP and c a
candidate action, we learn the parameters w and θ of a (state, command) value function of the form f(s̃, c) = 〈w,Ψθ(s̃, c)〉 where w ∈ Rd and Ψθ(s̃, c) is the output of a embedding network that maps (state, command) pairs to Rd, with parameters θ. In Q-learning and our algorithm presented in the next section, we directly use f as the state/action value function, whereas in policy gradient the probability to take command c in state s̃ is given by the Gibbs distribution over f(s̃, c) with temperature τ : π(c|s̃) = e
f(s̃,c)/τ∑ c′∈C e f(s̃,c′)/τ .
To tackle the last challenge, we identify units with their (x, y) coordinates in the map. We add two fields to the unit features that contain the coordinates of their corresponding target, or its own location if it does not have a target. To evaluate a command c = (<actor unit>, <attack or move>, <target>), we compute pairwise distances between the actor and the target. Note that with this kind of representation, the input of the embedding network Ψθ is a joint representation of the state s̃ and the command c to evaluate. A complete list of unit features is given in Table 1. Hit points are the remaining life points of the unit, shield corresponds to additional hit points that are not affected by armor and regenerate slowly, cooldown is the time to wait until damages can be inflicted.
The full scoring approach is depicted in Figure 1. In our approach, a state is represented as a list of units. The raw features are transformed by a featurizer that 1) takes the 3 unit features (pos, tgt_pos and next_pos) and computes their distances with the position the acting unit and its target (posc and tgtc). All 4 categorical variables are passed through a 10-dimensional linear embedding (not shown in figure). In addition to the 4 real valued unit features, we have a 40 dimensional feature vector per unit as input to our network.
Each unit feature vector then goes through the unit-level embedding network. We then concatenate the max and mean poolings across units with an embedding of the command type. Then, the resultant 210 dimensional vector is passed through a final state-command embedding network. Both the unit-level and state-command embedding networks have a hidden dimension of 100, and ELU nonlinearities in the intermediate layer (Clevert et al., 2015). We use tanh for the final unit-level network nonlinearty, and a ReLU for the final state-command network nonlinearity. We did not extensively experiment with the structure of the network, but we found the maxpooling and tanh nonlinearity to be particularly important.
The advantage of this approach is to rely on raw features only, and does not require any encoding of the game dynamics, in contrast to previous works on RL for micromanagement (see e.g. (Wender & Watson, 2012)) that used domain knowledge handcrafted in the features (such as the damages inflicted by an attack). The distance-based encoding is also a simple way to represent the different relationships between units that correspond to previous/chosen attacks.
6 COMBINING BACKPROPAGATION AND ZERO-ORDER OPTIMIZATION
Our preliminary experiments with Q-learning or REINFORCE made it clear that structured exploration was necessary to learn non-trivial strategies with substantial armies. The randomization of actions lead to the disorganization of the army and a rapid defeat, which prevents the algorithms from evaluating alterations to the current policy in the long run. Whereas gradient-free optimization that performs episode-based exploration (e.g. Mannor et al. (2003); Sehnke et al. (2010)) would be a valid choice, it only scales to few parameters. Preliminary experiments with direct exploration in the parameter space of the deep neural network confirmed that a more efficient scheme was needed.
The deterministic policy we consider takes action a in state s according to the rule
πw,θ(s) = argmax a∈A(s)
〈w,Ψθ(s, a)〉 .
We use the notation (s, a) for state and actions in an MDP for the presentation of the algorithm, even though in our experiments we use it with states s̃ of the greedy MDP and unit-level commands c. Likewise, we describe the algorithm in the standard cumulative reward setup, while in our experiments we use the normalized cumulative rewards.
This form of policy naturally allows to perform structured exploration by only randomizing parts of the network. More specifically, the parameters w of the last layer affect all states and actions in a similar way along an episode. The approach we follow is then to perform gradient-free optimization on these parameters w only. Following stochastic methods for zero-th order optimization (Kiefer et al., 1952; Nemirovsky et al., 1982; Spall, 1997; Duchi et al., 2013; Ghadimi & Lan, 2013), the gradient of a differentiable function x ∈ Rd 7→ f(x) can be estimated by
∇f(x) ≈ E[d δ f(x+ δu)u] ,
where the expectation is taken over the vector u sampled on the unit sphere (Nemirovsky et al., 1982, chapter 9.3). The constant dδ is absorbed by learning rates, so we ignore it in the following. Given a (state, action) pair (s, a) and the observed cumulative reward r̄1..t for an episode of length t, an estimate of the gradient of the expected cumulative reward with respect to w is thus r̄1..tu. In practice,
we use ∑t−1 k=1 r̄ k..t
t u rather than r̄ 1..tu, which corresponds to the gradient of the average cumulative
reward over the episode. We did not observe a large difference in preliminary experiments.
The overall algorithm is described in Algorithm 1. At the beginning of an episode, a perturbation u is sampled from the unit sphere of Rd and the policy s 7→ πw+δu,θ(s) is run through the entire episode (δ is a hyperparameter of the algorithm). The perturbation vector plays both role of performing structured exploration and providing the gradient estimate of the cumulative reward with respect to w. The algorithm performs a minibatch update at the end of the episode. The second loop in Algorithm 1 accumulates the update direction for w in ĝ(w) and the update direction for θ in Ĝ(Θ). The update
(*) ensures that ĝ(w) = ∑t−1 k=1 r̄ k..t
t u at the end of the loop, as described in the previous paragraph.
The deterministic exploration along an episode does not provide any update rule for the parameters of the embedding network, because the randomization is the same for every (state, action) pair. We propose a heuristic rule to update the parameters θ of the embedding network, motivated by the following remark: given a function (w ∈ Rd, v ∈ Rd) 7→ F (〈w, v〉) ∈ R, we have ∇wF = F ′(〈w, v〉)v and ∇vF = F ′(〈w, v〉)w. Denoting by wv the term-by-term division of vectors (assuming v contains only non-zero values) and the term-by-term multiplication operator, we obtain:
∇vF = (∇wF ) w
v .
Taking F to be the cumulative reward in (state, action) pair (sk, ak), assuming that it only depends on the dot product 〈w,Ψθ(sk, ak)〉, the update rule above gives ur̄k..t wΨθ(s,a) as input for backpropagation. In practice, we use the sign of wΨθ(s,a) to avoid exploding gradients due to the division by Ψθ(s, a), which seemed to work as well as more involved clipping heuristics. Line (**) of Algorithm 1 accumulates along the episode the updates for θ in Ĝ(θ). In (**), we divide the cumulative reward by t to optimize the average reward over the episode, and z 7→ backpropΨθ(sk,ak)(z) refers to the gradient with respect to θ when the network input is (sk, ak) and the backward step uses z as input.
Input exploration hyper-parameter δ, learning rate η, (state, action) embedding network Ψθ(s, a) taking values in Rd, with parameters θ ∈ Rm.; initialization: w ← 0, w ∈ Rd; while stopping criterion not met do
Sample u uniformly on the unit sphere of Rd; // follow the perturbated deterministic policy for one episode t← 0; while episode not ended do
t← t+ 1; observe current state st and reward rt; choose action at = argmaxa∈A(s)〈w + δ.u,Ψθ(st, a)〉;
end ĝ(w) = 0 // accumulation of minibatch update for w Ĝ(θ) = 0 ∈ Rm×d // accumulation of minibatch update for θ r̄t..t = 0 // cumulative reward, use n̄t..t = 0 for normalized rewards for k = t− 1 to 1 do
r̄k..t = r̄k+1..t + rk+1 // for normalized rewards, use n̄k..t and (1) ĝ(w)← ĝ(w) + r̄ k..t
t u (*); Ĝ(θ)← Ĝ(θ) + backpropΨθ(sk,ak)
( r̄k..t t u ( sign w
Ψθ(sk,ak)
)) (**);
end // perform gradient ascent update_adagrad(w, ηĝ(w)); update_adagrad(θ, ηĜ(θ));
end Algorithm 1: Zero-order (ZO) backpropagation algorithm
The reasoning above is only an intuitive motivation of the update rule (**) of Algorithm 1, because we neglected that a single u is sampled for an entire episode. We also neglected the argmax operation that chooses the actions. Nonetheless, considering (**) as a crude approximation to some real estimator of the gradient seems to work very well in practice, as we shall see in our experiments. Finally, we use Adagrad (Duchi et al., 2011) to update the parameters of the different layers. We found the use of Adagrad’s update scheme fairly important in practice, compared to other approaches such as e.g. RMSProp (Tieleman & Hinton, 2012), even though RMSProp tended to work slightly better with Q-learning or REINFORCE in our experiments.
7 EXPERIMENTS
7.1 SETUP
We use Torch7 (Collobert et al., 2011) for all our experiments. We connect our Torch code and models to StarCraft through a socket server, as described in (Synnaeve et al., 2016). We ran experiments with deep Q networks (DQN) (Mnih et al., 2013), policy gradient (PG) (Williams, 1992) (detailed in Appendix A), and zero order (ZO). We did an extensive hyper-parameters search, in particular over (for epsilon-greedy exploration in DQN), τ (for policy gradient’s softmax), learning rates, optimization methods, RL algorithms variants, and potential annealings (detailed Appendix E).
7.2 BASELINE HEURISTICS
As all the results that we report are against the built-in AI, we compare our win rates to the ones of baseline heuristics. Some of these heuristics often perform the micromanagement in full-fledged StarCraft bots (Ontanón et al., 2013), and are the basis of heuristic search (Churchill et al., 2012). The baselines are the following:
• random no change (rand_nc): select a random target for each of our units and do not change this target before it dies (or our unit dies). This spreads damage over several enemy units, but when there are collisions, it may make our units to move a lot to be in range of their target.
• noop: send no action. In this case, the built-in AI will control our units, so this exhibit the symmetry (or not!) of a given scenario. As we are always in a defensive position, with the enemy commanded to walk towards us, all other things considered equal, it should be easier for the defending built-in AI than for the attacking one. Our models cannot send a noop command.
• closest (c): each of our units targets the enemy unit closest to it. This is not a bad heuristic as enemy units formation will make it so that several of our units have the same opponent unit as closest unit (some form of focus firing), but not all of them (no overkill). It is also quite robust for melee units (e.g. Zealots) as it means they spend less time moving and more time attacking.
• weakest closest (wc): each of our units targets the weakest enemy unit. The distance of the enemy unit to the center of mass of our units is used for tie-breaking. This may overkill.
• no overkill no change (nok_nc): same as the weakest closest heuristic, but register the number of our units that target each opponent unit, choosing another target to focus fire when it becomes overkill to keep targeting a given unit. Each of our units keep firing on their target without changing (that would lead to erratic behavior). Our implementation of the “no overkill” component does not take all the dynamics of the game into account, and so if our units die without doing their expected damage on their target, “no overkill” can be detrimental.
7.3 RESULTS
The first thing that we looked at were sliding average win rates over 400 battles during training against the built-in AI of the various models. In Figure 2, we can see than DQN is much more dependent on initialization and variable than zero order (ZO). DQN can unlearn, reach suboptimal plateau, or overall need a lot of exploration to start learning (high sample complexity).
For all the results that we present in Tables 2 and 3, we ran the models in “test mode” by making them deterministic. For DQN we remove the epsilon-greedy exploration (set = 0), for PG we do not sample from the Gibbs policy but instead take the value-maximizing action, and for ZO we do not add noise to the last layer.
We can see in Table 2 that m15v16 is at the advantage of our player’s side (noop is at 81% win rate), whereas w15v17 is hard (c is at 20% win rate). By looking just at the results of the heuristics, we can see that overkill is a problem on m15v16 and w15v17 (nok_nc is better than wc). “Attack closest” (c) is approximatively as good as nok_nc at spreading damage, and thus better on m15v16 because there are lots of collisions (and attacking the closest unit is going to trigger less movements).
Overall, the zero order optimization outperforms both DQN and PG (REINFORCE) on most of the maps. The only map on which DQN and PG perform well is m5v5. It seems to be easier to learn
a focus firing heuristic (e.g. “attack weakest”) by identifying and locking on a feature, than to also learn not to “overkill”. We interpret the learned behaviors in Appendix F.
We then studied how well a model trained on one map performs on maps with a different number of units, to test generalization. Table 3 contains the results for this experiment. We observe that DQN performs the best on m5v5 when trained on m15v16, because it learned a simpler (but more efficient on m5v5) heuristic. “Noop” and “attack closest” are quite good with the large Marines map because they generate less moves (and less collisions). Overall, ZO is consistently significantly better than other RL algorithms on these generalization tasks, even though it does not reach an optimal strategy.
We also played the best model on each map against each other. We modify the maps in this case such that they are all symmetric, but with the same army composition. Table 4 shows the results for this experiment. It seems that PG and DQN learned very different strategies on wXvY, DQN beats PG consistently when trained on w15v17, while the PG model trained on w15v15 has an edge over DQN. Overall, ZO comes out ahead in every match-up except for m5v5, often by a significant margin.
8 CONCLUSION
This paper presents two main contributions. First, it establishes StarCraft micromanagement scenarios as complex benchmarks for reinforcement learning: with durative actions, delayed rewards, and large action spaces making random exploration infeasible. Second, it introduces a new reinforcement learning algorithm that performs better than prior work (DQN, PG) for discrete action spaces in these micromanagement scenarios, with robust training (see Figure 2) and episodically consistent exploration (exploring in the policy space).
This work leaves several doors open and calls for future work. Simpler embedding models of state and actions, and variants of the model presented here, have been tried, none of which produced efficient units movement (e.g. taking a unit out of the fight when its hit points are low). There is ongoing
work on convolutional networks based models that conserve the 2D geometry of the game (while embedding the discrete components of the state and actions). The zero order optimization technique presented here should be studied more in depth, and empirically evaluated on domains other than StarCraft (e.g. Atari). As for StarCraft scenarios specifically, the subsequent experiments will include self-play in training, multi-map training (more generic models), and more complex scenarios which include several types of advanced units with actions other than move and attack. Finally, the goal of playing full games of StarCraft should not get lost, so future scenarios would also include the actions of “recruiting” units (deciding which types of unit to use), and how to best make use of them.
ACKNOWLEDGEMENTS
We thank Y-Lan Boureau, Antoine Bordes, Florent Perronnin, Dave Churchill, Léon Bottou and Alexander Miller for helpful discussions and feedback about this work and earlier versions of the paper. We thank Timothée Lacroix and Alex Auvolat for technical contributions to our StarCraft/Torch bridge. We thank Davide Cavalca for his support on Windows virtual machines in our cluster environment.
A BASELINES
We here briefly describe the two algorithms we use as baseline, Q-learning (Sutton & Barto, 1998) and REINFORCE (Williams, 1992).
Q-learning The Q-learning algorithm in the finite-horizon setting learns an action-value function Q by solving the Bellman equation
∀s ∈ S,∀a ∈ A(s), Qt(s, a) = ∑ s′∈S ρ(s′|s, a) ( r(s, s′) + max a′∈A(s′) Qt+1(s ′, a′) ) , (2)
where Qt is the state-action value function at stage t of an episode, and QT (s, a) = 0 by convention. Qt(s, a) is also 0 whenever a terminal state is reached, and transitions from a terminal state only go to the same terminal state.
Training is usually carried out by collecting traces (st, at, st+1, rt+1)t=1,...,T−1 using -greedy exploration: at state s and stage t, an action in argmaxa∈A(s)Qt(s, a) is chosen with probability 1− , or an action in A(s) is chosen uniformly at random with probability . In practice, we use stationary Q functions (i.e., Qt = Qt+1), which are neural networks, as described in Section 5. Training is carried out using the standard online update rule for Q learning with function approximation (see (Mnih et al., 2015) for DQN), which we apply in mini-batches (hyper-parameters are detailed in Appendix E).
This training phase is distinct from the test phase, in which we record the average cumulative reward of the deterministic policy2 s 7→ argmaxa∈A(s)Q(s, a).
REINFORCE The algorithm REINFORCE belongs to the family of policy gradient algorithms (Sutton et al., 1999). Given a stochastic policy πΘ parameterized by Θ, learning is carried out by generating traces (st, at, st+1, rt+1)t=1,...,T−1 by following the current policy. Then, stochastic gradient updates are performed, using the gradient estimate:
T∑ t=1 r̄(st..T )∇Θ log(πΘ(at|st)) . (3)
We use a Gibbs policy (with temperature parameter τ ) as the stochastic policy:
πΘ(a|s) = exp(φΘ(a, s)/τ)∑
b∈A(s) exp(φΘ(b, s)/τ) ,
where φΘ is a neural network with paramters Θ that gives a real-valued score to each (state, action) pair. For testing, we use the deterministic policy πΘ(s) = argmaxa∈A(s) φΘ(a, s).
B THE GREEDY MDP
A natural way to define the greedy MDP (Section 4) is to define the set of atomic actions of the greedy policy as all possible (unit, command) pairs for the units whose command is still not decided. This would lead to an inference with quadratic complexity with respect to the number of units, which is undesirable.
We settled on iteratively choosing a unit, then a command to apply to that unit, which yields an algorithm with 2|s| steps for state s, linear in the number of units. Since the commands are executed concurrently by the environment after all commands have been decided, the cumulative reward does not depend on the order in which we choose the units, for instance: uniformly at random among remaining units. More formally, using the notation a1..k to denote the k first (unit, command) pairs of an action a (with the convention a1..0 = ∅), the state space S̃ of the greedy MDP is defined by
S̃ = { (s, a1..k, uk+1) ∣∣ s ∈ S, 0 ≤ k < |s|, a = ((u1, c1), ..., (u|s|, c|s|)) ∈ A(s)}.
2The policy may not be determistic if we break ties randomly in the argmax.
The action space A(s̃) of each state s̃ ∈ S̃ is constant and equal to the set of commands C. Moreover, for each state s of the original MDP, any action a = ((u1, c1), ..., (u|s|, c|s|) ∈ A(s), the transition probabilities ρ̃ in the greedy MDP are defined by
∀k ∈ {0, ..., |s| − 1}, ρ̃((s, a1..k, uk+1) ∣∣(s, a1..k−1, uk), ck) = 1|s| − k
and ∀s′ ∈ S,∀u′ ∈ U(s′), ρ̃((s′, ∅, u′) ∣∣(s, a1..|s|−1, u|s|), c|s|) = 1|s′|ρ(s′|s, a) .
Finally, using the same notation as above, the reward function r̃ between states that represent intermediate steps of the algorithm is 0 and the last unit to play receives the reward:
r̃((s, a1..k−1, uk), (s, a1..k, uk+1)) = 0 , and r̃((s, a1..|s|−1, u|s|), (s′, ∅, u′)) = r(s, s′) .
It can be shown that an optimal policy for this greedy MDP chooses actions that are optimal for the original MDP, because the immediate reward in the original MDP does not depend on the order in which the actions are taken. This result only applies if the family of policies has enough capacity. In practice, some ordering may be easier to learn than others, but we did not investigate this issue because the gain, in terms of computation time, of the random ordering was critical for the experiments.
C NORMALIZED CUMULATIVE REWARDS
The normalized rewards (from Section 4) maintain the invariant n̄t..T = r̄ t..T
z(st) ; but more importantly, the normalization can be applied to the Bellman equation (2), which becomes
∀s ∈ S,∀a ∈ A(s), Q(s, a) = ∑ s′∈S ρ(s′|s, a) z(s) ( r(s, s′) + z(s′) max a′∈A(s′) Q(s′, a′) ) .
This normalization does not change the optimal policy because it maintains the invariant that the expected normalized cumulative reward from a given state s to the end of an episode (by following the optimal deterministic policy) is the expected cumulative reward from this s divided by a value that depends only on s.
The stochastic gradient updates for Q-learning can easily be modified accordingly, as well as the gradient estimate in REINFORCE (3) in which we replace r̄ by n̄.
D STARCRAFT SPECIFICS
We advocate that using existing video games for RL experiments is interesting because the simulators are oftentimes complex, and we (the AI programmers) do not have control about the source code of the simulator. In RTS games like StarCraft, we do not have access to a simulator (and writing one would be a daunting task), so we cannot use (Monte Carlo) tree search (Gelly & Wang, 2006) directly, even less so in the setting of full games (Ontanón et al., 2013). In this paper, we consider the problem of micromanagement scenarios, a subset of full RTS play. Micromanagement is about making good use of a given set of units in an RTS game. Units have different features, like range, cooldown, hit points (health), attack power, move speed, collision box etc. These numerous features and the dynamics of the game advantage player that take the right actions at the right times. Specifically for the game(s) StarCraft, for which there are professional players, very good competitive players and professional players perform more than 300 actions per minute during intense battles.
We ran all our experiments on simple scenarios of battles of an RTS game: StarCraft: Broodwar. These scenarios can be considered small scale for StarCraft, but they already deem challenging for existing RL approaches. The joint action space is in Θ((#commands per unit)#units), with a peak number of units of about 400 (Synnaeve & Bessiere, 2011). For an example scenario of 15 units (that we control) against 16 enemy units, even while reducing the action space to "atomic" actions (surrounding moves, and attacks), we obtain 24 (8+16) possible discrete actions per unit for our controller to choose from (2415 actions total) at the beginning of the battle. Battles last for tens of seconds, with durative actions, simultaneous moves, and at 24 frames per second. The strategies that we need to learn consist in coordinated sets of actions that may need to be repeated, e.g. focus firing without overkill. We use a featurization that gives access only to the state from the game, we do not
pre-process the state to make it easier to learn a given strategy, thus keeping the problem elegant and unbiased.
For most of these tasks (“maps”), the number of units that our RL agent has to consider changes over an episode (a battle), as do its number of actions. The fact that we are playing in this specific adversarial environment is that if the units do not follow a coherent strategy for a sufficient amount of time, they will suffer an unrecoverable loss, and the game will be in a state of the game where the units will die very rapidly and make little damage, independently of how they play – a state that is mostly useless for learning.
Our tasks (“maps”) represent battles with homogeneous types of units, or with little diversity (2 types of unit for each of the players). For instance, they may use a unit of type Marine, that is one soldier with 40 hit points, an average move speed, an average range (approximately 10 times its collision size), 15 frames of cooldown, 6 of attack power of normal damage type (so a damage per second of 9.6 hit points per second, on a unit without armor). On symmetric and/or monotyped maps, strategies that are required to win (on average) are “focus firing”, without overkill (not more units targeting a unit than what is needed to kill it). For perfect win rates, some maps may require that the AI moves its units out from the focus firing of the opponent.
E HYPER-PARAMETERS
Taking an action on every frame (24 times per second at the speed at which human play StarCraft) for every unit would spam the game needlessly, and it would actually prevent the units from moving3. We take actions for all units synchronously on the same frame, even skip_frames frames. We tried several values of this hyper-parameter (5, 7, 9, 11, 13, 17) and we only saw smooth changes in performance. We ran all the following experiments with a skip_frames of 9 (meaning that we take about 2.6 actions per unit per second).We also report the strongest numbers for the baselines over all these skip frames. We optimize all the models after each battle (episode), with RMSProp (momentum 0.99 or 0.95), except for zero-order for which we optimized with Adagrad (Adagrad did not seem to work better for DQN nor REINFORCE). In any case, the learning rate was chosen among {10−2, 10−3, 10−4}. For all methods, we tried experience replay, either with episodes (battles) as batches (of sizes 20, 50, 100), or additionally with random batches of (st, at, rt+1, st+1, terminal?) quintuplets in the case of Q-learning, it did not seem to help compared to batching with the last battle. So, for consistency, we only present results where the training batches consisted of the last episode (battle).
For Q-learning (DQN), we tried two schemes of annealing for epsilon greedy, = 0√ 1+ a. 0.t with t the optimization batch, and = max(0.01, 0 a.t ), Both with 0 ∈ {0.1, 1}, and respectively a ∈ {0, 0} and a ∈ {10−5, 10−4, 10−3}. We found that the first works marginally better and used that in the subsequent experiments with 0 = 1 and a = 1 for most of the scenarios. We also used Double DQN as in (Van Hasselt et al., 2015) (thus implemented as target DQN). For the target/double network, we used a lag of 100 optimizations, thus a lag of 100 battles in all the following experiments. According to our initial runs/sweep, it seems to slightly help for some cases of over-estimation of the Q value.
For REINFORCE we searched over τ ∈ {0.1, 0.5, 1, 10}. For zero-order, we tried δ ∈ {0.1, 0.01, 0.001}.
F INTERPRETATION OF THE LEARNED POLICIES
We visually inspected the model’s performance on large battles. On the larger Marines map (m15v16), DQN learned to focus fire. Because this map has many units, focus firing leads to units bumping into each other to try to focus on a single unit. The PG player seemed to have a policy that attacks the closest marine, though it doesn’t do a good job switching targets. The Marines that are not in range often bump into each other. Our zero order optimization learns a hybrid between focus firing
3Because several actions are durative, including moves. Moves have a dynamic consisting of per-unit-type turn rate, max speed, and acceleration parameters.
and attacking the closest unit. Units would switch to other units in range if possible, but still focus on specific targets. This leads to most Marines attacking constantly, as well as focus firing when they can. However, the learned strategy was not perfected, since Marines would still split their fire occasionally when left with few units.
In the Wraiths map (w15v17), the DQN player’s strategy was hard to decipher. The most likely explanation is that they tried to attack the closest target, though it is likely the algorithm did not converge to a specific strategy. The PG player learned to focus fire. However, because it only takes 6 Wraiths to kill another, 9 actions are "wasted" during the focus firing (at the beginning of the fight, when all our units are alive). Our zero order player learns that focusing only on one enemy is not good, but it does not learn how many attacks are necessary. This leads to a much higher win rate, but the player still assigns more than 6 Wraiths to an enemy target (maybe for robustness to the loss of one of our units), and occasionally will not focus fire when only a few Wraiths are remaining. This is similar to what the zero order player learned during the Marines scenario.

The paper presents a learning algorithm for micromanagement of battle scenarios in real-time strategy games. It focuses on a complex sub-problem of the full RTS problem. The assumptions and restrictions made (greedy MDP, distance-based action encoding, etc.) are clear and make sense for this problem.
The main contribution of this paper is the zero-order optimization algorithm and how it is used for structured exploration. This is a nice new application of zero-order optimization meets deep learning for RL, quite well-motivated using similar arguments as DPG. The results show clear wins over vanilla Q-learning and REINFORCE, which is not hard to believe. Although RTS is a very interesting and challenging domain (certainly worthy as a domain of focused research!), it would have been nice to see results on other domains, mainly because it seems that this algorithm could be more generally applicable than just RTS games. Also, evaluation on such a complex domain makes it difficult to predict what other kinds of domains would benefit from this zero-order approach. Maybe the authors could add some text to clarify/motivate this.
There are a few seemingly arbitrary choices that are justified only by "it worked in practice". For example, using only the sign of w / Psi_{theta}(s^k, a^k). Again later: "Also we neglected the argmax operation that chooses the actions". I suppose this and dividing by t could keep things nicely within or close to [-1,1] ? It might make sense to try truncating/normalizing w/Psi; it seems that much information must be lost when only taking the sign. Also lines such as "We did not extensively experiment with the structure of the network, but we found the maxpooling and tanh nonlinearity to be particularly important" and claiming the importance of adagrad over RMSprop without elaboration or providing any details feels somewhat unsatisfactory and leaves the reader wondering why.. e.g. could these only be true in the RTS setup in this paper?
The presentation of the paper can be improved, as some ideas are presented without any context making it unnecessarily confusing. For example, when defining f(\tilde{s}, c) at the top of page 5, the w vector is not explained at all, so the reader is left wondering where it comes from or what its use is. This is explained later, of course, but one sentence on its role here would help contextualize its purpose (maybe refer later to the section where it is described fully). Also page 7: "because we neglected that a single u is sampled for an entire episode"; actually, no, you did mention this in the text above and it's clear from the pseudo-code too.
"perturbated" -> "perturbed"
--- After response period:
No rebuttal entered, therefore review remains unchanged.


CLASSLESS ASSOCIATION USING NEURAL NETWORKS
The goal of this paper is to train a model based on the relation between two instances that represent the same unknown class. This scenario is inspired by the Symbol Grounding Problem and the association learning in infants. We propose a novel model called Classless Association. It has two parallel Multilayer Perceptrons (MLP) that uses one network as a target of the other network, and vice versa. In addition, the presented model is trained based on an EM-approach, in which the output vectors are matched against a statistical distribution. We generate four classless datasets based on MNIST, where the input is two different instances of the same digit. In addition, the digits have a uniform distribution. Furthermore, our classless association model is evaluated against two scenarios: totally supervised and totally unsupervised. In the first scenario, our model reaches a good performance in terms of accuracy and the classless constraint. In the second scenario, our model reaches better results against two clustering algorithms.
1 INTRODUCTION
Infants are able to learn the binding between abstract concepts to the real world via their sensory input. For example, the abstract concept ball is binding to the visual representation of a rounded object and the auditory representation of the phonemes /b/ /a/ /l/. This scenario can be seen as the Symbol Grounding Problem (Harnad, 1990). Moreover, infants are also able to learn the association between different sensory input signals while they are still learning the binding of the abstract concepts. Several results have shown a correlation between object recognition (visual) and vocabulary acquisition (auditory) in infants (Balaban & Waxman, 1997; Asano et al., 2015). One example of this correlation is the first words that infants have learned. In that case, the words are mainly nouns, which are visible concepts, such as, dad, mom, ball, dog, cat (Gershkoff-Stowe & Smith, 2004). As a result, we can define the previous scenario in terms of a machine learning tasks. More formally, the task is defined by learning the association between two parallel streams of data that represent the same unknown class (or abstract concept). Note that this task is different from the supervised association where the data has labels. First, the semantic concept is unknown in our scenario whereas it is known in the supervised case. Second, both classifiers needs to agree on the same coding scheme for each sample pair during training. In contrast, the coding-scheme is already pre-defined before training in the supervised case. Figure 1 shows an example of the difference between a supervised association task and our scenario.
Usually, classifiers requires labeled data for training. However, the presented scenario needs an alternative training mechanism. One way is to train based on statistical distributions. Casey (1986) proposed to solve the OCR problem using language statistics for inferring form images to characters. Later on, Knight et al. (2006) applied a similar idea to machine translation. Recently, Sutskever et al. (2015) defined the Output Distribution Matching (ODM) cost function for dual autoencoders and generative networks.
In this paper, we are proposing a novel model that is trained based on the association of two input samples of the same unknown class. The presented model has two parallel Multilayer Perceptrons (MLPs) with an Expectation-Maximization (EM) (Dempster et al., 1977) training rule that matches the network output against a statistical distribution. Also, both networks agree on the same classification because one network is used as target of the other network, and vice versa. Our model has some
similarities with Siamese Networks proposed by Chopra et al. (2005). They introduced their model for supervised face verification where training is based on constraints of pairs of faces. The constraints exploit the relation of two faces that may or may not be instances of the same person. However, there are some differences to our work. First, our training rule does not have pre-defined classes before training, whereas the Siamese Network requires labeled samples. Second, our model only requires instances of the same unknown class, whereas the Siamese network requires two types of input pairs: a) instances of the same person and b) instances of two different persons. Our contributions in this paper are
• We define a novel training rule based on matching the output vectors of the presented model and a statistical distribution. Note that the output vectors are used as symbolic features similar to the Symbol Grounding Problem. Furthermore, the proposed training rule is based on an EM-approach and classified each sample based on generated pseudo-classes (Section 2.1).
• We propose a novel architecture for learning the association in the classless scenario. Moreover, the presented model uses two parallel MLPs, which require to agree on the same class for each input sample. This association is motivated by the correlation between different sensory input signals in infants development. In more detail, one network is the target of the other network, and vice versa. Also, note that our model is gradient-based and can be extended to deeper architectures (Section 2.2).
• We evaluate our classless association task against two cases: totally supervised and totally unsupervised. In this manner, we can verify the range of our results in terms of supervised and unsupervised cases since our model is neither totally supervised nor totally unsupervised. We compare against a MLP trained with labels as the supervised scenario (upper bound) and two clustering algorithms (K-means and Hierarchical Agglomerative) as the unsupervised scenario (lower bound). First, our model reaches better results than the clustering. Second, our model shows promising results with respect to the supervised scenario (Sections 3 and 4).
2 METHODOLOGY
In this paper, we are interested in the classless association task in the following scenario: two input instances x(1) and x(2) belong to the same unknown class c, where x(1) ∈ X(1) and x(2) ∈ X(2) are two disjoint sets, and the goal is to learn the output classification of x(1) and x(2) is the same c(1) = c(2), where c(1) and c(2) ∈ C is the set of possible classes. With this in mind, we present a model that has two parallel Multilayer Perceptrons (MLPs) that are trained with an EM-approach that associates both networks in the following manner: one network uses the other network as a target, and vice versa. We explain how the output vectors of the network are matched to a statistical distribution in Section 2.1 and the classless association learning is presented in Section 2.2.
2.1 STATISTICAL CONSTRAINT
One of our constraint is to train a MLP without classes. As a result, we use an alternative training rule based on matching the output vectors and a statistical distribution. For simplicity, we explain our training rule using a single MLP with one hidden layer, which is defined by
z = network(x;θ) (1)
where x ∈ Rn is the input vector, θ encodes the parameters of the MLP, and z ∈ Rc is the output vector. Moreover, the output vectors (z1, . . . ,zm) of a mini-batch of size m are matched to a target distribution (E[z1, . . . ,zm] ∼ φ ∈ Rc), e.g., uniform distribution. We have selected a uniform distribution because it is an ideal case to have a balanced dataset for any classifier. However, it is possible to extend to different distribution. We introduce a new parameter that is a weighting vector γ ∈ Rc. The intuition behind it is to guide the network based on a set of generated pseudo-classes c. These pseudo-classes can be seen as cluster indexes that group similar elements. With this in mind, we also propose an EM-training rule for learning the unknown class given a desired target distribution. We want to point out that the pseudo-classes are internal representations of the network that are independent of the labels.
The E-step obtains the current statistical distribution given the output vectors (z1, . . . ,zm) and the weighting vector (γ). In this case, an approximation of the distribution is obtained by the following equation
ẑ = 1
M M∑ i=1 power(zi,γ) (2)
where γ is the weighting vector, zi is the output vector of the network, M is the number of elements, and the function power1 is the element-wise power operation between the output vector and the weighting vector. We have used the power function because the output vectors (z1, . . . , zm) are quite similar between them at the initial state of the network, and the power function provides an initial boost for learning to separate the input samples in different pseudo-classes in the first iterations. Moreover, we can retrieve the pseudo-classes by the maximum value of the following equation
c∗ = arg maxc power(zi,γ) (3)
where c∗ is the pseudo-class, which are used in the M-step for updating the MLP weights. Also, note that the pseudo-classes are not updated in an online manner. Instead, the pseudo-classes are updated after a certain number of iterations. The reason is the network requires a number of iterations to learn the common features.
The M-step updates the weighting vector γ given the current distribution ẑ. Also, the MLP parameters (θ) are updated given the current classification given by the pseudo-classes. The cost function is the variance between the distribution and the desired statistical distribution, which is defined by
cost = (ẑ − φ)2 (4) 1We decide to use power function instead of zγi in order to simplify the index notation
where ẑ is the current statistical distribution of the output vectors, and φ is a vector that represent the desired statistical distribution, e.g. uniform distribution. Then, the weighting vector is updated via gradient descent
γ = γ − α ∗ ∇γcost (5)
where α is the learning rate and ∇γcost is the derivative w.r.t γ. Also, the MLP weights are updated via the generated pseudo-classes, which are used as targets in the backpropagation step.
In summary, we propose an EM-training rule for matching the network output vectors and a desired target statistical distribution. The E-Step generates pseudo-classes and finds an approximation of the current statistical distribution of the output vectors. The M-Step updates the MLP parameters and the weighting vector. With this in mind, we adapt the mentioned training rule for the classless association task. Figure 2 summarizes the presented EM training rule and its components.
2.2 CLASSLESS ASSOCIATION LEARNING
Our second constraint is to classify both input samples as the same class and different from the other classes. Note that the pseudo-class (Equation 3) is used as identification for each input sample and it is not related to the semantic concept or labels. The presented classless association model is trained based on a statistical constraint. Formally, the input is represented by the pair x(1) ∈ Rn1 and x(2) ∈ Rn2 where x(1) and x(2) are two different instances of the same unknown label. The classless association model has two parallel Multilayer Perceptron MLP (1) and MLP (2) with training rule that follows an EM-approach (cf. Section 2.1). Moreover, input samples are divided into several mini-batches of size m.
Initially, all input samples have random pseudo-classes c(1)i and c (2) i . The pseudo-classes have the same desired statistical distribution φ. Also, the weighting vectors γ(1) and γ(2) are initialized to one. Then each input element from the mini-batch is propagated forward to each MLP. Afterwards, an estimation of the statistical distribution for each MLP (ẑ(1) and ẑ(2)) is obtained. Furthermore, a new set of pseudo-classes (c(1)1 , . . . , c (1) m and c (2) 1 , . . . , c (2) m ) is obtained for each network. Note that this first part can be seen as an E-step from Section 2.1. We want to point out that the pseudo-classes are updated only after a number of iterations.
The second part of our association training updates the MLP parameters and the weighting vector (γ(1) and γ(2)). In this step, one network (MLP (1)) uses pseudo-classes (c(2)1 , . . . , c (2) m ) obtained from the other network (MLP (2)), and vice versa. In addition, the weighting vector is updated
between the output approximation (ẑ(1) and ẑ(2)) and the desired target distribution (φ). Figure 3 shows an overview of the presented model.
3 EXPERIMENTS
In this paper, we are interested in a simplified scenario inspired by the Symbol Grounding Problem and the association learning between sensory input signal in infants. We evaluated our model in four classless datasets that are generated from MNIST (Lecun & Cortes, 2010). The procedure of generating classless datasets from labeled datasets have been already applied in (Sutskever et al., 2015; Hsu & Kira, 2015). Each dataset has two disjoint sets input 1 and input 2. The first dataset (MNIST) has two different instances of the same digit. The second dataset (Rotated-90 MNIST) has two different instances of the same digit, and all input samples in input 2 are rotated 90 degrees. The third dataset (Inverted MNIST ) follows a similar procedures as the second dataset, but the transformation of the elements in input 2 is the invert function instead of rotation. The last dataset (Random Rotated MNIST) is more challenging because all elements in input 2 are randomly rotated between 0 and 2π. All datasets have a uniform distribution between the digits and the dataset size is 21,000 samples for training and 4,000 samples for validation and testing.
The following parameters turned out being optimal on the validation set. For the first three datasets, each internal MLP relies on two fully connected layers of 200 and 100 neurons respectively. The learning rate for the MLPs was set to start at 1.0 and was continuously decaying by half after every 1,000 iterations. We set the initial weighting vector to 1.0 and updated after every 1,000 iterations as well. Moreover, the best parameters for the fourth dataset were the same forMLP (1) and different for MLP (2), which has two fully connected layers of 400 and 150 neurons respectively and the learning rate stars at 1.2. The target distribution φ is uniform for all datasets. The decay of the learning rate (Equation 5) for the weighting vector was given by 1/(100+epoch)0.3, where epoch was the number of training iterations so far. The mini-batch size M is 5,250 sample pairs (corresponding to 25% of the training set) and the mean of the derivatives for each mini-batch is used for the back-propagation step of MLPs. Note that the mini-batch is quite big comparing with common setups. We infer from this parameter that the model requires a sample size big enough for estimating the uniform distribution and also needs to learn slower than traditional approaches. Our model was implemented in Torch.
MLP(1) MLP (2) Association Matrix (%) Purity (%)
10.9 10.9 __________Initial State
Epoch 1,000
0 1 2 3 4 5 6 7 8 9 MLP (2)
0 1 2 3 4 5 6 7 8 9 M LP (1 ) 0.7 0.6 2.4 0.4 3.0 2.2 1.0 1.4 1.2 0.6
0
2
4
6
8
10
12
14
16
18
20
24.8 22.6
Epoch 3,000
0 1 2 3 4 5 6 7 8 9 MLP (2)
0 1 2 3 4 5 6 7 8 9 M LP (1 ) 0.1 1.7 2.8 0.0 5.7 8.9 0.3 10.0 8.1 0.1
0
2
4
6
8
10
12
14
16
18
20
64.4 65.8
Epoch 49,000
0 1 2 3 4 5 6 7 8 9 MLP (2)
0 1 2 3 4 5 6 7 8 9 M LP (1 ) 9.8 8.9 9.4 9.0 9.6 9.6 9.7 9.6 9.6 9.0
0
2
4
6
8
10
12
14
16
18
20
95.5 95.6
Figure 4: Example of the presented model during classless training. In this example, there are ten pseudo-classes represented by each row of MLP (1) and MLP (2). Note that the output classification are randomly selected (not cherry picking). Initially, the pseudo-classes are assigned randomly to all input pair samples, which holds a uniform distribution (first row). Then, the classless association model slowly start learning the features and grouping similar input samples. Afterwards, the output classification of both MLPs slowly agrees during training, and the association matrix shows the relation between the occurrences of the pseudo-classes.
To determine the baseline of our classless constraint, we compared our model against two cases: totally supervised and totally unsupervised. In the supervised case, we used the same MLP parameters and training for a fair comparison. In the unsupervised scenario, we used k-means and agglomerative clustering to each set (input 1 and input 2) independently. The clustering algorithm implementation are provided by scikit-learn (Pedregosa et al., 2011).
4 RESULTS AND DISCUSSION
In this work, we have generated ten different folds for each dataset and report the average results. We introduce the Association Accuracy for measuring association, and it is defined by the following equation
Association Accuracy = 1
N N∑ i=1 1(c (1) i = c (2) i ) (6)
where the indicator function is one if c(1)i = c (2) i , zero otherwise; c (1) i and c (2) i are the pseudo-classes for MLP (1) and MLP (2), respectively, and N is the number of elements. In addition, we also reported the Purity of each set (input 1 and input 2). Purity is defined by
Purity(Ω, C) = 1 N k∑ i=1 maxj |ci ∩ gtj | (7)
where Ω = {gt1, gt2, . . . , gtj} is the set of ground-truth labels and C = {c1, c2, . . . , ck} is the set of pseudo-classes in our model or the set of cluster indexes of K-means or Hierarchical Agglomerative clustering, and N is the number of elements.
Table 1 shows the Association Accuracy between our model and the supervised association task and the Purity between our model and two clustering algorithms. First, the supervised association task performances better that the presented model. This was expected because our task is more complex in relation to the supervised scenario. However, we can infer from our results that the presented model has a good performance in terms of the classless scenario and supervised method. Second, our model not only learns the association between input samples but also finds similar elements covered under the same pseudo-class. Also, we evaluate the purity of our model and found that the performance of our model reaches better results than both clustering methods for each set (input 1 and input 2).
Figure 4 illustrates an example of the proposed learning rule. The first two columns (MLP (1) and MLP (2)) are the output classification (Equation 3) and each row represents a pseudo-class. We have randomly selected 15 output samples for each MLP (not cherry picking). Initially, the pseudo classes are random selected for each MLP. As a result, the output classification of both networks does not show any visible discriminant element and the initial purity is close to random choice (first row). After 1,000 epochs, the networks start learning some features in order to discriminate the input samples. Some groups of digits are grouped together after 3,000 epochs. For example, the first row of MLP (2) shows several digits zero, whereas MLP (1) has not yet agree on the same digit for that pseudo-class. In contrast, both MLPs have almost agree on digit one at the fifth row. Finally, the association is learned using only the statistical distribution of the input samples and each digit is represented by each pseudo-class.
Best Results
Worst Results
MLP (1) MLP (2) Association Matrix (%)
0 1 2 3 4 5 6 7 8 9 MLP (2)
0 1 2 3 4 5 6 7 8 9 M LP (1 ) 9.7 8.9 9.0 8.7 9.7 9.0 9.2 9.1 9.4 8.5
0
2
4
6
8
10
12
14
16
18
20
Purity (%)
0 1 2 3 4 5 6 7 8 9 MLP (2)
0 1 2 3 4 5 6 7 8 9 M LP (1 ) 9.7 7.9 5.1 4.5 0.0 5.1 9.2 0.0 10.3 12.8
0
2
4
6
8
10
12
14
16
18
20
95.9 95.2
72.9 59.4
Figure 5: Example of the best and worst results among all folds and datasets. It can be observed our model is able to learn to discriminate each digit (first row). However, the presented model has a limitation that two or more digits are assigned to the same pseudo-class (last row of MLP (1) and MLP (2)).
Figure 5 shows the best and worst results of our model in two cases. The first row is the best result from MNIST dataset. Each row of MLP (1) and MLP (2) represent a pseudo-class, and it can be observed that all digits are grouped together. In addition, the association matrix shows a distribution per digit close to the desired uniform distribution, and the purity of each input is close to the supervised scenario. In contrast, the second row is our worst result from Random Rotated MNIST dataset. In this example, we can observe that some digits are recognized by the same pseudo-class, for example, digit one and seven (first two rows). However, there two or more digits that are recognized by the samepseudo-class. For example, the last row shows that nine and four are merged. Our model is still able to reach better results than the unsupervised scenario.
5 CONCLUSION
In this paper, we have shown the feasibility to train a model that has two parallel MLPs under the following scenario: pairs of input samples that represent the same unknown classes. This scenario was motivated by the Symbol Grounding Problem and association learning between sensory input signal in infants development. We proposed a model based on gradients for solving the classless association. Our model has an EM-training that matches the network output against a statistical distribution and uses one network as a target of the other network, and vice versa. Our model reaches better performance than K-means and Hierarchical Agglomerative clustering. In addition, we compare the presented model against a supervised method. We find that the presented model with respect to the supervised method reaches good results because of two extra conditions in the unsupervised association: unlabeled data and agree on the same pseudo-class. We want to point out that our model was evaluated in an optimal case where the input samples are uniform distributed and the number of classes is known. However, we will explore the performance of our model if the number of classes and the statistical distrubtion are unknown. One way is to change the number of pseudo-classes. This can be seen as changing the number of clusters k in k-means. With this in mind, we are planning to do more exhaustive analysis of the learning behavior with deeper architectures. Moreover, we will work on how a small set of labeled classes affects the performance of our model (similar to semi-supervised learning). Furthermore, we are interested in replicating our findings in more complex scenarios, such as, multimodal datasets like TVGraz (Khan et al., 2009) or Wikipedia featured articles (Rasiwasia et al., 2010). Finally, our work can be applied to more classless scenarios where the data can be extracted simultaneously from different input sources at the same time. Also, transformation functions can be applied to input samples for creating the association without classes.
ACKNOWLEDGMENTS
We would like to thank Damian Borth, Christian Schulze, Jörn Hees, Tushar Karayil, and Philipp Blandfort for helpful discussions.

The paper explores a new technique for classless association, a milder unsupervised learning where we do not know the class labels exactly, but we have a prior about the examples that belong to the same class. Authors proposed a two stream architecture with two neural networks, as streams process examples from the same class simultaneously. Both streams rely on the target (pseudo classes or cluster indices) of each other, and the outputs an intermediate representation z, which is forced to match with a statistical distribution (uniform in their case). The model is trained with EM where the E step obtains the current statistical distribution given output vectors z, and M step updates the weights of the architecture given z and pseudo-classes. Experimental results on re-organized MNIST exhibits better performance compared to classical clustering algorithms (in terms of association accuracy and purity). The authors further provide comparison against a supervised method, where proposed architecture expectedly performs worse but with promising results.
The basic motivation of the architecture apparently relies on unlabeled data and agreement of the same pseudo-labels generated by two streams. But the paper is hard to follow and the motivation for the proposed architecture itself, is hidden in details. What is trying to be achieved by matching distributions and using the pseudo-targets of the each other? Perhaps the statistical distribution of the classes is assumed to be uniform but how will it extend to other priors, or even the case where we do not assume that we know the prior? The current setup needs justifications.
What would be very interesting is to see two examples having the same class but one from MNIST, the other from Rotated-MNIST or Background-MNIST. Because it is hard to guess how different the examples in two streams.
At the end, I feel like the authors have found a very interesting approach for classless association which can be extended to lots of many-to-one problems. This is a good catch. I would like to see the idea in the future with some extensive experiments on large scale datasets and tasks. But the current version lacks the theoretical motivations and convincing experiments. I would definitely recommend this paper to be presented in ICLR workshop.
Few more points:
Typo: Figure1. second line in the caption "that" -> "than"
Necessity of Equation 2 is not clear
Batch size M is enormous compared to classical models, there is no explanation for this
Why uniform? should be clarified (of course it is the simplest prior to pick but just a few words about it would be good for completeness)
Typo: Page 6, second paragraph line 3: "that" -> "than"