Linear Feature Encoding for Reinforcement Learning
Feature construction is of vital importance in reinforcement learning, as the quality of a value function or policy is largely determined by the corresponding features. The recent successes of deep reinforcement learning (RL) only increase the importance of understanding feature construction. Typical deep RL approaches use a linear output layer, which means that deep RL can be interpreted as a feature construction/encoding network followed by linear value function approximation. This paper develops and evaluates a theory of linear feature encoding. We extend theoretical results on feature quality for linear value function approximation from the uncontrolled case to the controlled case. We then develop a supervised linear feature encoding method that is motivated by insights from linear value function approximation theory, as well as empirical successes from deep RL. The resulting encoder is a surprisingly effective method for linear value function approximation using raw images as inputs.
This paper addresses the problem of designing and learning models for planning in reinforcement learning. More specifically, it proposes a linear model comprising of an "encoder", mapping the observation space into a latent space, and a "decoder", mapping the latent space to the predicted next state and reward. The encoder/decoder model draws inspiration from autoencoders (linear) but couples transition dynamics and reward under the same model. The authors use Parr's (2007) Bellman error decomposition to analyze the their model. A factorization approach is chosen for learning such models using the SVD. Experiments finally show that this model can be used in high dimensional settings when learning directly from pixels.
Summary:  The idea of coupling reward and dynamics in an autoencoder-like model is a novel contribution which could benefit our community. I also appreciate that the authors have applied their model on pixel-based observation spaces. However, I find the theory of lines 124 to 136 unnecessary and the fact that it reproduces Parr (2007) line by line is problematic (more on this below). Also, example 1 seems misguided since it simply does adopt the right problem formulation to start with (it seems sufficient to simply start with a Markov chain over state-action pairs).  Detailed comments:  Abstract: l. 4. and sect. 1 l.25: "Typical deep RL [...]" and "It is common"  Is that true ? Beside DQN, what are other examples ? Is it true for TD-Gammon ?  section 1, l. 21: "researchers at Deepmind". I would use a different form. Simply citing the DQN paper would probably be enough.  sect 1. l. 31: The qualifier "novel" seems unnecessary.  sect 1. l. 41: "Aside from the negative one". We don't know yet at this point what this mean. Perhaps you can reference the section in which the negative one is presented.  section 1. l.42 : "Persistent hidden state". This is not clear what you mean here. Perhaps you meant to compare your model-based approach with more of an RNN or PSR-type model. If this is the case, you would have to argue that the proposed model is somehow able to learn a representation which makes the observations as "Markov" as possible, that you can "make state" without state/belief state updating.  section 1. l. 47: "Our results differ from". Please cite the work implied in this sentence. Perhaps Parr (2007), Sutton (2008) ?  section 2. l. 56: This is not any MDP, it's a *Discounted* MDP. Why do you need to define the reward function over $(n \times m) \times 1$ ?  section 2. l. 62: "The policy $\pi$": This is a particular kind of policy, namely a "stochastic" or "randomized" policy.  section 2. l. 65: Sum: "Expected total *sum* of $\gamma$-discounted"  section 2.1 l. 71: "Otherwise, we have [...]. This sentence lacks context. It should probably be mentioned that the Bellman operator is a contraction and that you solve for the fixed point through value iteration.  section 2.3 : "has typically": please cite relevant literature. "Feature selection wrapper": "wrapper" is more of a term that you would use in software engineering. The meaning is not clear in this context.  section 2.3 l. 86: Please explain why it did not scale well.  section 2.3 l. 88: "connect directly to the optimization problem". This isn't clear. Knowing this paper, I think that you would want to say is that PVFs only consider the transition dynamics and not the reward structure.  section 2.3 l. 91: "classical approaches". What are they ? Please cite relevant work.  section 2.3 l. 98: it could be relevant to also mention "Value Pursuit Iteration" by Farahmand and Precup.  section 3. l. 109: "Anecdotally" generally connotes with "amusing", which is not the intended meaning here.  section 3. l. 106-108: I find this sentence confusing because it mixes concepts of "action-conditional" linear models (which is more about the "evaluation problem") with the "control problem" (policy/value iteration). One does not imply the other.  section 3. l. 109. "The following example [...]". I find this sentence confusing and it sounds in a way that contradics Parr (2007). Also, you need to define more formally what you mean by "linear action models" earlier in the section.  Lines 124 to 136 (and section 3.0 as a whole) seem unnecessary. The derivations follow exactly line by line (and using the same notation) that of Parr (2007). It would suffice to say that in order to represent Q-values, we need to consider the augmented Markov chain over state-action pairs. You define the state $\tilde{s}_t = (s_t, a_t)$ and then simply use Parr's results over this chain. The same argument applies when you go from TD(0) to SARSA(0) : see Sutton & Barto for example https://webdocs.cs.ualberta.ca/~sutton/book/ebook/node64.html  section 3. following definition 5: A novel aspect of the proposed approach is that it couples reward and transition dynamics under the same predictor. I think that it's an important design choice for which it would be worth spending more time explaining. You could for example contrasts this approach with one based exclusively on the transitions dynamics (Mahadevan's PVFs for example).  section 3.1, l. 170: Theorem numbering jumps from 3 to 7.  section 4, l. 208: "termination conditions are typical API [...]", please briefly explain why they still hold.  section 4.1 : Please explain why you chose coordinate descent. Also, it would be useful to study the effect of the estimation error in $\tilde{A}$ on the quality of the learned features. Is it stable ?  Lines 220-223 is useful to know but it is also more difficult to clearly see what is the main algorithm proposed in this section. It could perhaps be moved to the appendix.  section 5, l. 246: "Ambitious". Many examples of large scale linear RL can be found in the literature on the Arcade Learning Environment (prior to DQN). For example, see "State of the Art Control of Atari Games Using Shallow Reinforcement Learning" by Liang, Machado & al. for recent work.  section 5, l. 249: I would want to see this claim back up in your experiments: "far greater computational resources". The iterative SVD approach of algorithm 2. with O(n^3) per iteration is non-negligible. It is also not clear that it could benefit from the same computational speedups as GPU-based deep RL methods.  section 5, l. 257: half an hour with or without the exploration phase ?  section 5.1, l. 277: I find this experiment very interesting. However, if the simulator had to be modified, it suggests that the proposed approach cannot cope with the non-Markovianity by building appropriate features.  Have you tried concatenating a number of successive frames (4) as in DQN ?  section 5.1, l. 286: how did you choose $k$ ?  section 5.1: It would be important to remind the reader that you use the planning approach of algorithm 1 in this experiment. You could have also chosen to learn the model using algorithm 2 and plan using any other approach.  section 5.2, l. 321: Why did you have to define this metric rather than reporting the actual return ?  section 5.2, l. 347: What was this threshold ?  How did you choose it ?