Deep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization. Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning. These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difficulty of the game can be tuned by changing environment parameters in an interpretable way. We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but also to compare approaches based on supervised and reinforcement learning, to analyze the power of multi-agent approaches in improving performance, and to evaluate generalization to environments outside the training set.
The paper presents Erdos-Selfridge-Spencer games as environments for investigating
deep reinforcement learning algorithms. The proposed games are interesting and clearly challenging, but I am not sure what they tell us about the algorithms chosen to test them. There are some clarity issues with the justification and evaluation which undermine the message the authors are trying to make.
In particular, I have the following concerns:
  • these games have optimal policies that are expressible as a linear model, meaning that if the architecture or updating of the learning algorithm is such that there is a bias towards exploring these parts of policy space, then they will perform better than more general algorithms. What does this tell us about the relative merits of each approach? The authors could do more to formally motivate these games as "difficult" for any deep learning architecture if possible.
  • the authors compare linear models with non-linear models at some point for attacker policies, but it is unclear whether these linear models are able to express the optimal policy. In fact, there is a level of non-determinism in how the attacker policies are encoded which means that an optimal policy cannot be (even up to soft-max) expressed by the agent (as I read things the number of pieces chosen in level l is always chosen uniformly randomly).
  • As the authors state, this paper is an empirical evaluation, and the theorems presented are derived from earlier work. There is possibly too much focus on the proofs of these theorems.
  • There are a number of ambiguities and errors which places difficulties on the interpretation (and potential replication) of the experiments. As this is an empirical study, this is the yardstick by which the paper should be judged. In particular, this relates to:
    ◦ The architecture of each of the tested Deep RL methods.
    ◦ What is done to select appropriate tuning parameters of the tested Deep RL methods, if anything.
    ◦ It is unclear whether 'incorrect actions' in the supervised learning evaluations, refer to non-optimal actions, or simply actions that do not preserve the dominance of the defender, e.g. both partitions may have potential >0.5
    ◦ Fig 4. right looks like a reward signal, but is labelled Proportion correct. The text is not clear enough to be sure which it is.
    ◦ Fig 4. left and right has 4 methods: rl rewards, rl correct actions, sup rewards, and sup correct actions. The specifics of how these methods are constructed is unclear from the paper.
    ◦ What parts of the evaluation explores how well these methods are able to represent the states (feature/representation learning) and what parts are evaluating the propagation of sparse rewards (the reinforcment learning core)? The authors could be clearer and more targetted with respect to this question.
There is value in this work, but in its current state I do not think it is ready for publicaiton.
# Detailed notes
[p4, end of sec 3] The authors say that the difficulty of the games can be varied with "continuous changes in potential", but the potential is derived from the discrete initial game state, so these values are not continuously varying (even though it is possible to adjust them by non-integer amounts).
[p4, sec 4.1]
"strategy unevenly partitions the occupied levels...with the proportional difference between the two sets being sampled randomly"
What is meant by this? The proportional difference between the two sets is discussed as if it is a continuous property, but must be chosen from the discrete set of all available partitions. If one partition one is chosen uniformly randomly from all possibly sets A, B (and the potential proportion calculated) then I don't know why it would be written in this way. That suggests that proportions that are closer to 1:1 are chosen more often than "extreme" partitions, but how? This feels a little under-justified.
"very different states A, B (uneven potential, disjoint occupied levels)"
Are these states really "very different", or at least for the reasons indicated. Later on (Theorem 3) we see how an optimal partition is generated. This chooses a partition where one part contains all pieces in layer (l+1) and above and one part with all pieces in layer (l-1) and below, with layer l being distributed between the two parts. The first part will typically have a slightly lower potential than the other and all layers other than layer l will be disjoint.
[p6, Fig 4] The right plot y-limits vary between -1 and 1 so it cannot represent a proportion of correct actions. Also, in the text the authors say:
  >> The results, shown in Figure 4 are surprising. Reinforcement learning 
  >> is better at playing the game, but does worse at predicting optimal moves.
I am not sure which plot shows the playing of the game. Is this the right hand plot? In which case are we looking at rewards? In fact, I am a little confused as to what is being shown here. Is "sup rewards" a supervised learning method trained on rewards, or evaluated on rewards, or both? And how is this done. The text is just not clear enough.
[p7 Fig 6 and text] Here the authors are comparing how well agents select the optimal actions as compared to how close they are to the end of the game. This relates to the "surprising" fact that "Reinforcement learning is better at playing the game, but does worse at predicting optimal moves.". I think an important point here is how many training/test examples there are in each bin. If there are more in the range 3-7 moves from the end of the game, than there are outside this range, then the supervised learner will
[p8 proof of theorem 3] 
"φ(A l+1 ) < 0.5 and φ(A l ) > 0.5."
Is it true that both these inequalities are strict?
"Since A l only contains pieces from levels K to l + 1"
In fact this should read from levels K to l.
"we can move k < m − n pieces from A l+1 to A l"
Do the authors mean that we can define a partition A, B where A = A_{l+1} plus some (but not all) elements in level l (A_{l}\setminus A_{l+1})?
"...such that the potential of the new set equals 0.5"
It will equal exactly 0.5 as suggested, but the authors could make it more precise as to why (there is a value n+k < l (maybe <=l) such that (n+k)*2^{-(K-l+1)}=0.5 (guaranteed). They should also indicate why this then justifies their proof (namely that phi(S0)-0.5 >= 0.5).
[p8 paramterising action space] A comment: this doesn't give as much control as the authors suggest. Perhaps the agent should also chose the proportion of elements in layer l to set A. For instance, if there are a large number of elements in l, and or phi(A_{l+1}) is very close to 0.5 (or phi(A_l) is very close to 0.5) then this doesn't give the attacker the opportunity to fine tune the policy to select very good partitions.  It is unclear expected level of control that agents have under various conditions (K and starting states).
[p9 Fig 8] As the defender's score is functionally determined by the attackers score, it doesn't help to include this on the plot. It just distracts from the signal.