To gain high rewards in muti-agent scenes, it is sometimes necessary to understand other agents and make corresponding optimal decisions. We can solve these tasks by first building models for other agents and then finding the optimal policy with these models. To get an accurate model, many observations are needed and this can be sample-inefficient. What's more, the learned model and policy can overfit to current agents and cannot generalize if the other agents are replaced by new agents. In many practical situations, each agent we face can be considered as a sample from a population with a fixed but unknown distribution. Thus we can treat the task against some specific agents as a task sampled from a task distribution. We apply meta-learning method to build models and learn policies. Therefore when new agents come, we can adapt to them efficiently. Experiments on grid games show that our method can quickly get high rewards.
This paper focuses on fast adaptation to new behaviour of the other agents of the environment, be it opponents or allies. To achieve this, a method based on MAML is proposed, with two main components:
1) Learn a model of some characteristics of the opponent, such as "the final goal, next action, or any other character we wish to predict"
2) Learn a policy that takes as input the output of the model and the state, and that outputs the action of the agent.
The goal is that after a phase of meta learning, where the agents learns how to play against some new agents sampled from the distribution of opponents, it can quickly adapt to a new unseen agent. ("Experimental results show that the agent can adapt to the new opponent with a small number of interactions with the opponent")
While the motivation of this work is clear and the goal important for the RL community, the experiments fail to support the claim above.
The first task they demonstrate their approach on is a chasing game, where the opponent has a private goal cell it tries to reach, and the agent has to chase it. At the end of the game, it gets a reward of 10 if it is on the same cell, 5 if in an adjacent cell, and 0 otherwise. The exact details of the dynamic are not really clear, for example what happens in the event of a collision is not mentionned, and the termination condition is not mentionned either. (the text reads "One game lasts for at least 15 steps", maybe it was meant to be "at most 15 steps" ?).
The first incoherent aspect of this experiment is that they use 800 iterations of meta-learning, and then, when testing, they fine-tune their networks against each test opponent during 4000 games. That is, they use 5 times more game when fine-tuning as opposed to when pre-training, which contradicts the claim "the agent can adapt to the new opponent with a small number of interactions with the opponent" (this is not really few-shot learning anymore).
Further more, they compare their approach with various ablations of it: they either remove the meta-learning for the model (MA), for the policy (MO), or both (NM). The description of the NM baseline is not very precise, but it seems that it simply boils down to a regular (dueling) DQN: In this setting, since the opponent appears to have a fixed goal, finetuning against a single opponent simply boils down to learning a policy that reaches a specific cell of the grid, which we can expect DQN to solve perfectly on a 8x8 grid with 4000 training games. And yet, the curves for NM in graph 2c is not only really noisy, but also falls far from the optimum, which the authors don't discuss. There might be a problem with the hyperparameters used or the training loop.
The second task is a blocking game: the opponent has to choose amongst 5 paths to get to the top, and the agent has to choose the same path in order to block it. The action space should be precisely described, as it stands it is difficult to understand the dynamic. There are at least two possible ways to parametrize the actions:
1) Similarly to the blocking game, the agents could move in the 8 directions. In that case, based on the picture 3a, it seems that the agent can just mirrors the move of the opponent: since the moves are simultaneous, that would mean that the agent is always one step late, but each path is long enough for the agent to reach the exit before its opponent (it explicitly stated that the agent needs to block the exit, and that the opponent will not change path during one game). That would imply that perfect play is possible without any meta-learning or oponent modeling, and once again the NM baseline (or any vanilly DQN/Policy gradient method) should perform much better.
2) One other alternative is to have an action space of 5 actions, which correspond to the 5 paths. In that case the game boils down to a bandit, since both agents only take one action. Note that under this assumption, the random policy would get the right path (and reward +10) with probability 1/5 and a wrong one (reward -10) with probability 4/5, which leads to an expectated reward of -10*4/5 + 10/5 = -6. This is not consistent with the graph 3c, since at the beginning of the training, the NM agent should have a random policy, and yet the graph reports an average reward of -10 (the -6 mark seems to be reached after ~1000 episodes)
The last task boils down to one opponent that reaches one cell on the right, and the agent must reach the matching cell on the left. In this setting, the same discussion on the action space as the second task can be made. We note that the episode for 16 steps, and the distance from the center to any cell is at most 4 steps: an optimal policy would be to wait for 4 steps in the middle, and as soon as the opponent has reached its goal, use the remaining 12 steps to get to the mirror one. Once again, this policy doesn't require any prediction on the opponent's goal, and it's hard to believe that DQN (possibly with an lstm) is not able to learn that near perfectly.
In a last test the authors compare the performance of their algorithms in a one shot transfer setting: they sample 100 opponents for each task and play only one game against it (no fine-tuning). It is not clear whether special care has been taken to ensure that none of the sampled opponents has already been seen during training.
We note that the rewards reported for MO and MA (resp 0.0 and -0.08) are not consistent with the description of the reward function: on the worst case, the opponent chooses a goal on one extreme (say y1 = 1) and the agent chooses an object on the other end (say y2 = 7). In that case, the reward obtained is sampled from a gaussian with mean \mu = 10 - 3/2 * |y1 - y2| (which in this case evalutes to 1), and variance 1. This is highly unlikely to give such a low average reward over 100 episodes (note that this is worst case, if the opponent's goal is not on the extreme, the expected reward is necessarily higher). One possibility is that the agent never reaches an object, but in that case it would imply the that the meta-learning phase was problematic.
We also note that it is explicited that the MOA, MO and MA methods are tested after meta-training, but nothing is precised for NM. Has it been trained at all? Against which opponents? Is it just a random policy? There are too many missing details for the results to be interpretable.
Apart from that, the paper contains a significant amount of typos and gramatical mistakes please proof-read carefully. Some of them are:
"To demonstrate that meta-learning can do take"
"player 1 is the red grid and player 1 is the green one"
"we further assume that there exist a distribution"
" the goal’s location over the map is visualize in figure"
"Both players takes actions simultaneously"