Contrary to most reinforcement learning studies, which emphasize on training a deep neural network to approximate its output layer to certain strategies, this paper proposes a reversed method for reinforcement learning. We call this "Deep Deducing". In short, after adequately training a deep neural network according to a strategy-environment-to-payoff table, then we initialize randomized strategy input and propagate the error between the actual output and the desired output back to the initially-randomized strategy input in the "input layer" of the trained deep neural network gradually to perform a task similar to "human deduction". And we view the final strategy input in the "input layer" as the fittest strategy for a neural network when confronting the observed environment input from the world outside.
The paper proposes a method of searching for a Nash equilibrium strategy in games where the strategy-to-payoff mapping is defined by a neural network. The idea is to perform gradient optimization of the payoff w.r.t. the strategy. Preliminary results on tic-tac-toe and variations of the prisoner’s dilemma task are presented. The paper has an interesting idea at the core. However, it is poorly written, does not properly discuss the related works and does not present a convincing method or experimental results. 
Pros:
* The paper considers an interesting question of exploring the applications of neural networks to the game theory problems.
* The idea of the paper is reasonable. It makes sense to me to perform gradient-based search over the strategies (assuming that the payoff is differentiable).
Cons:
* Writing
- The paper is over the mandatory length limit of 10 pages.
- The paper makes a grandiose claim: “this paper provides a revolutionary way for reinforcement learning and a possible road toward general A.I.” However, there are arguably no revolutionary ideas, and certainly no reinforcement learning experiments!
- Despite the claim, the novelty of the paper is limited. There is no discussion of the related work: optimization of the neural networks w.r.t. the inputs [1]; related RL ideas such as model-based learning [2,3] and Monte-Carlo Tree Search [4].
- The problem being solved is never formally stated. As far as I understand, Nash equilibrium is usually defined (1) in mixed strategies, while the paper seems to consider pure strategies; (2) in the scenario where every player attempts to maximize their payoff, while in the paper the players attempt to achieve some pre-fixed value of the payoff.
- The flow of the paper is generally poor. Instead of presenting a general solution and then showcasing its applications, the paper iterates on similar ideas multiple times. For example, all four algorithms are just gradient-based optimization of either the weights or the inputs to a model.
- The paper provides extremely misleading analogies and explanations. I am quite sure that a mosquito brain is not a one hidden layer fully-connected neural network! Also, the example of avoiding a moving hand is poor: since the outcome is life or death, the learning should happen via evolution, not during the lifetime of a single insect. The claim that the neural networks with sigmoid activation functions are less prone to local optima is questionable as well.
* Method and experiments
- The proposed method is essentially a greedy gradient-based planning procedure. For this to work, we need to have a very good environment model. This is a strong assumption that is not discussed.
- The experiments are performed on very simple synthetic problems: matrix games and tic-tac-toe. They do not suggest that the method is general and can work on harder problems, say, Sokoban [2].
- The experiments do not present any baselines, so it is unclear how well the method performs compared to the alternatives. One obvious candidate is gradient-free optimization, such as Nelder-Mead, and gradient descent with momentum, which can be less prone to local optima.
[1] Brandon Amos, Lei Xu, J. Zico Kolter “Input Convex Neural Networks”, ICML 2017
[2] Racanière et al. “Imagination-Augmented Agents for Deep Reinforcement Learning”, NIPS 2017
[3] David Ha, Jürgen Schmidhuber “Recurrent World Models Facilitate Policy Evolution”, NIPS 2018
[4] Thomas Anthony, Zheng Tian, David Barber “Thinking Fast and Slow with Deep Learning and Tree Search”, NIPS 2017