Bayesian Control of Large MDPs with Unknown Dynamics in Data-Poor Environments
We propose a Bayesian decision making framework for control of Markov Decision Processes (MDPs) with unknown dynamics and large, possibly continuous, state, action, and parameter spaces in data-poor environments. Most of the existing adaptive controllers for MDPs with unknown dynamics are based on the reinforcement learning framework and rely on large data sets acquired by sustained direct interaction with the system or via a simulator. This is not feasible in many applications, due to ethical, economic, and physical constraints. The proposed framework addresses the data poverty issue by decomposing the problem into an offline planning stage that does not rely on sustained direct interaction with the system or simulator and an online execution stage. In the offline process, parallel Gaussian process temporal difference (GPTD) learning techniques are employed for near-optimal Bayesian approximation of the expected discounted reward over a sample drawn from the prior distribution of unknown parameters. In the online stage, the action with the maximum expected return with respect to the posterior distribution of the parameters is selected. This is achieved by an approximation of the posterior distribution using a Markov Chain Monte Carlo (MCMC) algorithm, followed by constructing multiple Gaussian processes over the parameter space for efficient prediction of the means of the expected return at the MCMC sample. The effectiveness of the proposed framework is demonstrated using a simple dynamical system model with continuous state and action spaces, as well as a more complex model for a metastatic melanoma gene regulatory network observed through noisy synthetic gene expression data.
POST-REBUTTAL:  Thank you for the clarifications. I've increased the overall score because the rebuttal made me think of this work as an interesting tradeoff between generality and the time when when computational cost investment has to be made (online vs. offline). At the same time, BAMCP has the significant advantage of handling more stochastic (as opposed to nearly deterministic, as in this paper) scenarios, and I would encourage you to work on extending your approach to them.   ==============================================================  The paper proposes a Bayesian RL method for MDPs with continuous states, action, and parameter spaces. Its key component is efficiently approximating the Q-value function expectations over the posterior belief over the parameter space, which is achieved by a combination of  GPTD during the offline stage of the algorithm and MCMC during the online stage. The approach is evaluated against MMRL and greedy myopic action selection on two problems, one of which is a synthetic MDP and another is a problem of computing gene expression interventions for a regulatory network.  To me, the paper's main novelty is the technique for approximating Q-value function expectations over parameter posterior given approximate Q-values for a sample of parameter values drawn from the prior. The approach has quite a few moving parts, and it's impressive to see them work in concert.  At the same time, the paper would benefit greatly from a better explanation of its motivation/applicability and, on a related note, from crisper positioning w.r.t. related work. The lack of clarity in these aspects makes it difficult to determine potential existing alternatives (if any) to the proposed algorithm and its parts. More specifically:  - Motivation. The submission's promise of handling decision-making problems with uncertain dynamics and continuous parameter spaces in data-poor environments isn't unique to the approach presented here. It is the motivation for Bayesian RL in general, and different BRL methods deliver on it varying degrees of success. The question is: what exactly distinguishes the proposed approach's capabilities from prior art in BRL? Is it the ability to handle continuous state spaces? This is a bit hard to believe, because given a sample of model parameters, i.e., a fully specified MDP, there is a range of methods that can solve such MDPs; discretizing the state space is one option. Same goes for continous action spaces. In fact, I don't see why this paper's method would have particular advantages in this regard, because despite its ability to approximate Q-value expectations, it doesn't offer anything special to facilitate action selection based on these Q-value estimates -- a non-trivial problem in itself when you use critic methods in continuous state spaces. In short, from the paper's current description, I couldn't deduce the advantages of the proposed method (or the problem characteristics where its advantages are especially noteworthy).  - Positioning w.r.t. related work. Currently, the paper seems to mention only one three BRL works, and they are really dated by now. Exactly what other algorithms the proposed work should be compared to depends on its motivation, but one algorithm that is almost certainly relevant is BAMCP:  Guez, Silver, Dayan. "Scalable and Efficient Bayes-Adaptive Reinforcement Learning Based on Monte-Carlo Tree Search". JAIR, 2013.  Its vanilla version described above assumes discrete state and action spaces, but using discretization this restriction can be lifted, and its derivatives may have done so already. In general, there has been a lot of BRL work since references [7], [8], and [9], as described in the following survey:  Ghavamzadeh, Mannor, Pineau, Tamar. "Bayesian Reinforcement Learning: A Survey". Foundations and Trends in Machine Learning, 2015. Newer versions are on arXiV.  It would be good to see the related work coverage expanded to at least some of the recent BRL methods described there.    More technical comments:  - The phrase "uncertain dynamics" is somewhat unfortunate, because it can be easily misinterpreted to refer to dynamics that are probabilistic, while this paper seems to handle only the cases where the dynamics itself is near-deterministic but unknown.  - The claim on lines 108-113 makes Q-value computation for a known \theta looks harder than it is. Every known \theta essentially encodes an MDP with known dynamics (and known reward function -- this appears to be assumed in the paper). Calculating the Q-function for this MDP is a planning problem, which can be solved in a wide variety of ways, which can be very efficient depending on the MDP structure. In particular, I'm not sure why deep learning is necessary here. Is this because you assume continuous states and actions and a DNN-based Q-value approximator?  - Is Equation 3 missing the expectation sign? If not, it is missing something else, because, as stated, the value of the left-hand side depends on a particular trajectory sampled from \pi, and the LHS value is ill-defined without conditioning on this trajectory.  - I'm confused by Section 3.1. Its stated goal is computing (an approximation of) Q^*_{\theta}, the Q-value function of the optimal policy under MDP parameters \theta. However, then it goes on to explain how to compute Q^{\pi}_{\theta} for policy \pi from Equation 10. What's the relationship between \pi from Equation 10 and the optimal policy under \theta? Line 147 talks about the exploration-exploitation tradeoff -- why do you need to resolve it if you are doing all of this for a _specific_ MDP parameter vector \theta?  - The paper switches between subscript and superscript indexes for \theta. It appears to be using subscript indexes for \thetas sampled from the prior (as on line 175) and superscript indexes for \thetas sampled from the posterior (as in Equation 14). Is this the distinction the paper is trying to make? If so, I suggest keeping it uniform and using, e.g. \theta^{prior}_i and \theta^{posterior}_i instead.  - Are there any hypotheses as to why the proposed algorithm outperforms MMRL on the synthetic MDP?  - To beef up the the experimental evaluation, I'd replace the synthetic MDP with something more complicated.   Language errors:  "the sequence of observed joint state and action simulated" --> "the sequence of observed joint state and action pairs simulated"  "specially in regions" --> "especially in regions"   IN THE REBUTTAL, please comment on the motivation/advantages of the proposed algorithms w.r.t. others, particularly BAMCP and its derivatives, and cover the questions in the "technical comments" section of the review, as space allows.