Counterfactual Regret Minimization (CRF) is a fundamental and effective technique for solving Imperfect Information Games (IIG). However, the original CRF algorithm only works for discrete state and action spaces, and the resulting strategy is maintained as a tabular representation. Such tabular representation limits the method from being directly applied to large games and continuing to improve from a poor strategy profile. In this paper, we propose a double neural representation for the imperfect information games, where one neural network represents the cumulative regret, and the other represents the average strategy. Furthermore, we adopt the counterfactual regret minimization algorithm to optimize this double neural representation. To make neural learning efficient, we also developed several novel techniques including a robust sampling method, mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and Monte Carlo Counterfactual Regret Minimization Plus (MCCFR+) which may be of independent interests. Experimentally, we demonstrate that the proposed double neural algorithm converges significantly better than the reinforcement learning counterpart.
========= Summary =========
The authors propose "Double Neural CFR", which uses neural network function approximation in place of the tabular update in CFR. CFR is the leading method for finding equilibria in imperfect information games. However it is typically employed with a tabular policy, limiting its applicability large games. Typically, hand-crafted abstractions are employed for games that are too large for exact tabular solutions. Function approximation could remove the necessity for hand-crafted abstractions and allow CFR to scale to larger problems.
The DN-CFR algorithm roughly consists of:
- start with an arbitrary vmodel_0, smodel_0
for t = 0,1,2,3:
  - collect a "batch" of (infoset I, immediate counterfactual value v_I) samples by traversal against vmodel_t (as well as I, strategy samples)
  - train a new network vmodel_{t+1} on this "batch" with y=(v_I + vmodel_t(I)) and MSE loss
  - similarly for (I, strategy)
- return smodel_t
The authors also propose a novel MC samping strategy this is a mixture between outcome and external sampling.
DN-CFR is evaluated on two games: a variant of Leduc hold-em with stack size 5, and one-card poker with 5 cards. If I understand correctly, these games have <10,000 and <100 infosets, respectively. The authors show that DN-CFR achieves similar convergence rates to tabular CFR, and outperform NFSP variants.
========== Comments ========
The authors are exploring an important problem that is of great interest to the IIG community. Their application of NN function approximation is reasonable and mostly theoretically well-grounded (but see below), I think it's on the right track. However, the games that are used for evaluation are very small, in fact I believe they have fewer states than the number of parameters in their network (the number of network parameters is not provided but I assume >1000). As a result, the NN is not providing any compression or generalization, and I would expect that the network can memorize the training set data exactly, i.e. predict the exact mean counterfactual value for each infoset over the data. If that's true, then DN-CFR is essentially exactly replicating tabular CFR (the approximation serves no purpose). 
As a result, in my opinion this work fails to address the important challenges for function approximation in CFR, namely:
- Can function approximation allow for *generalization across infosets* in order to reduce sample complexity of CFR (i.e. an unsupervised abstraction)? Are specific network architectures required for good generalization?
- The magnitude of counterfactual regrets in the support of the equilibrium decays to zero relative to dominated actions. Are NN models capable to estimating the regrets accurately enough to converge to a good strategy?
- Are optimization methods able to deal with the high variance in large IIGs?
- Since each successive approximation is being trained from the previous NN, does this cause errors to accumulate?
- How do approximation errors accumulate across CFR iterations?
- Is minimizing MSE loss sufficient to approximate the strategy well? (since the mapping from regrets -> strategy is non-linear)
I believe there is also a theoretical problem with this algorithm. In Eq. 8, they minimize the loss of CF value predictions *over the distribution of infosets in the last CFR step ("batch")*. However, this distribution may change between CFR iterations, for example if the strat_t folds 2-3 on the preflop then flop infosets with 2-3 hole cards will never be observed on iteration t+1. As a result, the NN loss will only be minimized over infosets observed in the last iteration - so the network will "forget" the regrets for all other infosets. I think this issue does not arise in these toy games because all infosets are observed at each iteration, but this is certainly not the case in real games. 
There are a number of ways that this issue could be addressed (e.g. train on historical infosets rather than the current batch, etc.) These would need to be explored.
I would recommend that the authors evaluate on more complex games to answer the important questions stated above and resubmit. I think this would also make an excellent workshop submission in its current state as it contains many interesting ideas.
Detailed comments:
"... the original CFR only works for discrete stand and action spaces...": Are the authors implying that DN-CFR addresses this limitation? 
"Moravk" -> "Moravcik"
"...these methods do not explicitly take into account the hidden information in a game..." Could you clarify? Is your point that these methods operate on the normal form rather than extensive form game?
"care algorithm design" -> "careful algorithm design"
The paragraph starting "In standard RNN..." should be reduced or moved to appendix. The exact NN architecture is not central to the ideas, and there are no experimental comparison with other architectures so we have no evidence that the architecture is relevant.