Surprise-based learning allows agents to adapt quickly in non-stationary stochastic environments . Most existing approaches to surprise-based learning and change point detection assume either implicitly or explicitly a simple, hierarchical generative model of observation sequences that are characterized by stationary periods separated by sudden changes. In this work we show that exact Bayesian inference gives naturally rise to a surprise-modulated trade-off between URLetting and integrating the new observations with the current belief. We demonstrate that many existing approximate Bayesian approaches also show surprise-based modulation of learning rates, and we derive novel particle filters and variational filters with update rules that exhibit surprise-based modulation. Our derived filters have a constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these filters estimate parameters better than alternative approximate approaches and reach comparative levels of performance  to computationally more expensive algorithms. The  theoretical insight of casting various approaches under the same interpretation of surprise-based learning, as well as the proposed filters, may find useful applications in reinforcement learning in non-stationary environments and in the analysis of animal and human behavior  .
Surprise-based learning allows agents to rapidly adapt to non-stationary stochastic environments characterized by stationary periods separated by sudden changes. We show that exact Bayesian inference in a hierarchical model gives rise to a surprise-modulated trade-off between URLetting old observations and integrating them with the new ones. The modulation depends on a probability ratio, called the "Bayes Factor Surprise" that tests the prior belief against the current belief. We demonstrate that in several existing approximate algorithms the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprised-based algorithms, one in the family of particle filters, one in the family of variational learning, and the other in the family of message passing, that are biologically plausible, have constant scaling in observation sequence length and particularly simple update dynamics for any distribution in the exponential family. Empirical results show that these surprise-based algorithms estimate parameters better than alternative approximate approaches and reach levels of performance comparable to computationally more expensive algorithms. The Bayes Factor Surprise is related to but different from Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological or behavioral indicators that dissociate the Bayes Factor Surprise from Shannon Surprise. The theoretical insight of casting various approaches as surprise-based learning, as well as the proposed online algorithms, may be applied to the analysis of animal and human behavior , and to reinforcement learning in non-stationary environments .
R
adapt quickly in
rapidly adapt to
D
. Most existing approaches to surprise-based learning and change point detection assume either implicitly or explicitly a simple, hierarchical generative model of observation sequences that are
R
In this work we
We
R
gives naturally
in a hierarchical model gives
R
and integrating the new observations with the
old observations and integrating them with the new ones. The modulation depends on a probability ratio, called the "Bayes Factor Surprise" that tests the prior belief against the
R
many existing approximate Bayesian approaches also show surprise-based modulation of learning rates, and we derive novel particle filters and variational filters with update rules that exhibit surprise-based modulation. Our derived filters have a
in several existing approximate algorithms the Bayes Factor Surprise modulates the rate of adaptation to new observations. We derive three novel surprised-based algorithms, one in the family of particle filters, one in the family of variational learning, and the other in the family of message passing, that are biologically plausible, have
R
filters
surprise-based algorithms
D
comparative
A
comparable
A
Bayes Factor Surprise is related to but different from Shannon Surprise. In two hypothetical experiments, we make testable predictions for physiological or behavioral indicators that dissociate the Bayes Factor Surprise from Shannon Surprise. The
R
under the same interpretation of
as
R
filters, may find useful applications in reinforcement learning in non-stationary environments and in
online algorithms, may be applied to
A
, and to reinforcement learning in non-stationary environments