Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm -an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.
The paper concerns a stochastic variational approach towards mutual information maximisation with applications to reinforcement learning. The authors present a treatment of MI following a bound taken form earlier work by Barber&Agakov and using it stochastically by mini-batch descent. In order to estimate the MI they introduce a novelty and use neural networks to predict parameters for factors in state transition models as shown in Equation 6. This replaces the need for an explicit generative model of the data. The authors use this algorithm in the context of empowerment, which is a measure that can be used to connect MI with reinforcement learning. they demonstrate very simple empowerment based policies using their MI approximation which yield experimentally good results while running much more efficiently than previous algorithms used to estimate the MI.The paper is written clearly and appears to be of good quality throughout. The organization is slightly perplexing, as frequently the purpose of the paper is mixed up between the efficient estimation of MI and reinforcement learning and Iwould urge the authors to clarify what their precise goals are: the paper is not sufficiently focused on reinforcement learning to be evaluated as such and it also fails to clearly show broad applicability for stochastic variational MI in non-reinforcement learning settings.That said, the results appear encouraging and efficient flexible methods for modeling actions in environments using neural networks are currently relevant directions for research.Comparisons with Auto-Encoding Variational Bayes would be interesting, as the approximation of factors using neural networks in a variational setting is reminiscent of this paper. I would also be interested in seeing how exactlythe performance of the model depends on the length of the action states given that the variational stochastic approximation to the gaussian factors does not use any kind of control variates to help learning of these factors and the fact that a factorization over many variational distributions will become worse and worse as the number of the factors grows without modeling interactions of components.