LEARNING FINITE STATE REPRESENTATIONS OF RECURRENT POLICY NETWORKS
Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.
This paper proposes a method to learn a quantization of both observations and hidden states in an RNN. Its findings suggest that many problems can be reduced to relatively simple Moore Machines, even for complex environments such as Atari games.
The method works by pretraing an RNN to learn a policy (e.g. through the A3C algorithm), and then training pairs of encoder/decoder networks with a quantizing forward pass and a straight-through backpropagation. The learned quantizations can then be used to build a Moore Machine, which itself can be reduced with FSM reduction algorithms, yielding a discrete, symbolic approximation of the inner workings of RNNs, that could in principle be interpreted more easily than latent embedding spaces.
One downside of this paper is that it promises an exciting method to analyse the inner workings of RNNs, but then postpones this analysis to later work. Understandably, the synthetic experiments take some space and shows that the proposed method works as expected when the problem is amenable to discretization; maybe some parts of this could be in the appendix?
Another downside is that there is little indication of the computational implications of the method. The method was evaluated on a fairly small set of hyperparameters, and there are no indication of how long the optimization and finetuning takes. Presumably, minimizing a Moore Machine has been studied for decades, but how long does minimizing the 1000s of states in Atari games take? A second or an hour?
The paper is fairly well written and easy to understand. The method seems well grounded, although I'm not familiar enough with the quantization literature to detect if something important is missing. I think this is a great tool that hopefully will be used to try to understand the memory mechanisms of RNNs. 
I think the proposed method (and the fact that it works in simple cases) warrants acceptance, but I think more experimental work would make this a great contribution. Since there is no reason for quantization to improve performance if it is done after training, then more emphasis should be put on the interpretability of the discretization; yet it is lacking in the current work. Some Atari games are known to require various amounts of memory, this could be analysed. Some other Atari games are known to be hard to solve, what happens to the RNN when the agent fails to achieve an optimal policy might also show up in the subsequent discretization and be interesting to analyse.
Comments:
- In atari, you can have access to the RAM and from it, using exactly the same mechanisms and maybe a bit of tabular MDPs, you should be able to recover the optimal MM.
- It is good that the authors report their failure to train MMNs from scratch; IMO this says something about the straight through estimators' limits. Measuring how sensible these things are to change in their target distribution and comparing to previous uses of ST in quantization works could be interesting.
- in Section 8 (appendix) "Grammer" should be "Grammar"
- All the (PO)MDPs that you analyse arguably have finite state spaces, and you set the ALE to be deterministic. What happens in continuous stochastic environments? 
- Do you think a similar technique could be used to recover a (possibly stochastic) MDP instead of a Moore Machine? It would be interesting to see MDP reduction methods applied to a learned MDP.