PAC Reinforcement Learning with Rich Observations
We propose and study a new model for reinforcement learning with rich observations, generalizing contextual bandits to sequential decision making. These models require an agent to take actions based on observations (features) with the goal of achieving long-term performance competitive with a large set of policies.
To avoid barriers to sample-efficient learning associated with large observation spaces and general POMDPs, we focus on problems that can be summarized by a small number of hidden states and have long-term rewards that are predictable by a reactive function class. In this setting, we design and analyze a new reinforcement learning algorithm, Least Squares Value Elimination by Exploration. We prove that the algorithm learns near optimal behavior after a number of episodes that is polynomial in all relevant parameters, logarithmic in the number of policies, and independent of the size of the observation space. Our result provides theoretical justification for reinforcement learning with function approximation.
This is a technical paper which proves PAC learnability of a complex model, but it does not make a convincing case that the model is a useful extension of standard MDPs.
The paper analyzes PAC learnability of finite-horizon Contextual MDPs. Contextual MDPs are a specific type of POMDPs with the restriction that the optimal q-function depends only on the most recent observation (instead of the belief state). The authors show that Contextual MDPs are not poly PAC learneable even when either memoryless policies are considered or value function approximation is used. However, when both memoryless policies and value function approximation is used and the transitions are deterministic, then the model is PAC learnable in a polynomial number of episodes (and the complexity is independent of the number of observations).  The paper is well written overall. The proofs are quite clear and quite thorough. I am not quite sure that the 16 pages of technical proofs in the appendix are suitable for a conference; the paper may better fit a journal format. The presentation could be further improved by better describing the results and how they were obtained earlier on in the paper. It takes a lot of effort to really understand the contribution and what are the main ideas in constructing the new algorithm.  The main limitation of the paper is that its results apply to a rather limited setting. While it is clear that Contextual MDPs are slightly more general than plain MDPs, the authors do not make a convincing case that any interesting or useful problems fall into this category. In particular the results apply to a POMDP in which 1) transitions are deterministic, 2) optimal q-function depends only on the most recent observation, and 3) the value function approximator can exactly represent the optimal q-function.  The authors only spend a single paragraph on the justification of why this may be a useful model in Example 1, but the example is quite construed.  There are very few (if any at all) plausible types of POMDPs in which the optimal q-value depends only on the last observation and action that are not also MDPs. I could imagine a model in which the optimal policy is reactive (memoryless), but the requirement of the value function to be memoryless is just too strong.   The technical result of the paper is interesting in that it proposes a sampling method that is independent of the number of observations. It is a generalization of learnability results for MDP (with the caveat that the transitions are deterministic), but the proof and algorithms are much less elegant.   Perhaps is the paper was posed in terms of learnability of MDPs with value function approximation, the  results would be more meaningful and applicable.  More detailed comments: 1) Introduction: the first few paragraphs are quite vague, the references (e.g. [21]) do not really support the claims, and the paper does not at all answer the question posed. 2) The layered process is really just a standard finite-horizon MDP (with non-stationary transitions) 3) Appendix A: Please start with the description of what is in the appendix. One needs to read the entire Theorem 2 to understand that it is just a technical lemma needed to prove Prop 1 & 2. 4) Section 2.2, Definition 1: This is a bit repetitive given the description in 2.1 5) Assumption 1: The requirement to have only deterministic transitions limits the applicability of the results quite a bit 6) Section 3.1: The algorithms and basic ideas of the proof are described very nicely in this section. 7) References have several instances of incorrect capitalization 8) Line 404: The argument regarding no need to consider stochastic policies is not convincing. I guess the equivalence follows from “splitting policies in MDPs”, but it would be good to provide more details.