Body: <p>I'm trying to implement the paper <em><a href="https://ieeexplore.ieee.org/document/9708819" rel="nofollow noreferrer">A Curiosity Algorithm for Robots Based on the Free Energy Principle</a></em> in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is calculated.</p>
  <p>As the paper describes, I made a 'transitioner' which predicts the next state given the current state and action. One layer of that transitioner is a Bayesian linear layer from <a href="https://github.com/piEsposito/blitz-bayesian-deep-learning" rel="nofollow noreferrer">Blitz</a>; the probability distribution of weights in that layer is <span class="math-container">$q_ψ(w) = N(w|μ,σ_2)$</span> where <span class="math-container">$ψ = \{μ,σ\}$</span>. Then, curiosity is defined as the Kullback–Leibler divergence <span class="math-container">$D_{KL}[q_ψ(w|s_{t+1})||q_ψ(w)]$</span>.</p>
  <p><span class="math-container">$q_ψ(w)$</span> is easily available, but <span class="math-container">$q_ψ(w|s_{t+1})$</span> is described as value of <span class="math-container">$q$</span> minimizing free energy <span class="math-container">$F = D_{KL}[q_ψ(w)||p_ψ(w)] − E_{q_ψ}(w)[log (p_ψ(s_{t+1}|w))]$</span>, and <span class="math-container">$p_ψ$</span> isn't given. I think I understand the concept, but I'm puzzled how to implement this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-20T02:24:54.127Z
Comments.Text: To update: I am learning much by focusing on the paper inspiring Blitz: "Weight Uncertainty in Neural Networks" (also cited in "A Curiosity Algorithm for Robots Based on the Free Energy Principle"). It seems the loss-function for weight probability distributions is, in fact, the variational free energy. Hence, DKL[qψ(w|st+1)||qψ(w)] literally describes comparing weights before and after training.
Comments.UserDisplayName: Ted Tinker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-11T13:06:02.970Z
Comments.Text: You can use mathjax on this site.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-12T05:33:50.260Z
Comments.Text: Oh, thank you. I'll try editing the question.
Comments.UserDisplayName: Ted Tinker
Tags: <reinforcement-learning><machine-learning><probability-distribution><artificial-curiosity><bayesian-reinforcement-learning>
Title: Calculating Curiosity with Friston's Free Energy in Reinforcement Learning
UserDisplayName: Ted Tinker
ThreadText: # Calculating Curiosity with Friston's Free Energy in Reinforcement Learning
  
  (Asked by: username_0 on 2022-11-18)
  
  I'm trying to implement the paper *[A Curiosity Algorithm for Robots Based on the Free Energy Principle](https://ieeexplore.ieee.org/document/9708819)* in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is calculated.
  
  As the paper describes, I made a 'transitioner' which predicts the next state given the current state and action. One layer of that transitioner is a Bayesian linear layer from [Blitz](https://github.com/piEsposito/blitz-bayesian-deep-learning); the probability distribution of weights in that layer is $q\_ψ(w) = N(w|μ,σ\_2)$ where $ψ = \{μ,σ\}$. Then, curiosity is defined as the Kullback–Leibler divergence $D\_{KL}[q\_ψ(w|s\_{t+1})||q\_ψ(w)]$.
  
  $q\_ψ(w)$ is easily available, but $q\_ψ(w|s\_{t+1})$ is described as value of $q$ minimizing free energy $F = D\_{KL}[q\_ψ(w)||p\_ψ(w)] − E\_{q\_ψ}(w)[log (p\_ψ(s\_{t+1}|w))]$, and $p\_ψ$ isn't given. I think I understand the concept, but I'm puzzled how to implement this.
  
  **Tags:** reinforcement-learning, machine-learning, probability-distribution, artificial-curiosity, bayesian-reinforcement-learning
  > **Comments:**
  > **username_0**: To update: I am learning much by focusing on the paper inspiring Blitz: "Weight Uncertainty in Neural Networks" (also cited in "A Curiosity Algorithm for Robots Based on the Free Energy Principle"). It seems the loss-function for weight probability distributions is, in fact, the variational free energy. Hence, DKL[qψ(w|st+1)||qψ(w)] literally describes comparing weights before and after training.  
  > **username_1**: You can use mathjax on this site.  
  > **username_0**: Oh, thank you. I'll try editing the question.  
  
  ---
  
  ## 0 answers
  
  