Information-based learning by agents in unbounded state spaces
The idea that animals might use information-driven planning to explore an unknown environment and build an internal model of it has been proposed for quite some time. Recent work has demonstrated that agents using this principle can efficiently learn models of probabilistic environments with discrete, bounded state spaces. However, animals and robots are commonly confronted with unbounded environments. To address this more challenging situation, we study informationbased learning strategies of agents in unbounded state spaces using non-parametric Bayesian models. Specifically, we demonstrate that the Chinese Restaurant Process (CRP) model is able to solve this problem and that an Empirical Bayes version is able to efficiently explore bounded and unbounded worlds by relying on little prior information.
In their paper “Information-based learning by agents in unbounded state spaces” the authors extend a previous model of information-based exploration as described in reference [11] to unbounded state spaces by introducing a Chinese restaurant process to model transition probabilities.Previous studies have used the Chinese restaurant process for reinforcement learning—for example, reference [2] cited by the authors. It would therefore be good if the authors could clarify the differences to previous studies that have used Chinese restaurant processes in reinforcement learning to clarify originality.L 130: verb is missing in the second part of the sentenceTo compute the information gain the authors need to compute relative entropies between the true state transition distribution and the estimated state transition distribution. As explained by the authors, these two distributions initially have different dimensionality. To make the distributions commensurable they expand the dimensionality of the estimated distribution to include states that have not been observed to date by uniform distributions. However, this seems to defeat the purpose of the paper to some extent, since the true cardinality of all random variables has to be known in equation (9) and all subsequent equations (10)-(13). This is a crucial question to clarify.L 160Question wrt equation (12)The last \hat{\theta}_sa is the updated model? In that case shouldn’t it be written as \hat{\theta}^{a,s\rightarrow s^*} as in equation (6) in reference [11]?The information gain measure the authors use in equation (13) is somewhat unconventional. Therefore, it would be important to see how it performs in the most simple exploration problem—that is the bandit problem. This should be even simpler to simulate than the experiments they present. For bandit problems there are well established bounds on how efficient different exploration methods are and the authors could compare against established schemes like UCB or Thompson sampling.