<p>I'm trying to implement the paper <em><a href="https://ieeexplore.ieee.org/document/9708819" rel="nofollow noreferrer">A Curiosity Algorithm for Robots Based on the Free Energy Principle</a></em> in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is calculated.</p>
<p>As the paper describes, I made a 'transitioner' which predicts the next state given the current state and action. One layer of that transitioner is a Bayesian linear layer from <a href="https://github.com/piEsposito/blitz-bayesian-deep-learning" rel="nofollow noreferrer">Blitz</a>; the probability distribution of weights in that layer is <span class="math-container">$q_ψ(w) = N(w|μ,σ_2)$</span> where <span class="math-container">$ψ = \{μ,σ\}$</span>. Then, curiosity is defined as the Kullback–Leibler divergence <span class="math-container">$D_{KL}[q_ψ(w|s_{t+1})||q_ψ(w)]$</span>.</p>
<p><span class="math-container">$q_ψ(w)$</span> is easily available, but <span class="math-container">$q_ψ(w|s_{t+1})$</span> is described as value of <span class="math-container">$q$</span> minimizing free energy <span class="math-container">$F = D_{KL}[q_ψ(w)||p_ψ(w)] − E_{q_ψ}(w)[log (p_ψ(s_{t+1}|w))]$</span>, and <span class="math-container">$p_ψ$</span> isn't given. I think I understand the concept, but I'm puzzled how to implement this.</p>

CC BY-SA 4.0
2022-11-20T02:24:54.127Z
To update: I am learning much by focusing on the paper inspiring Blitz: "Weight Uncertainty in Neural Networks" (also cited in "A Curiosity Algorithm for Robots Based on the Free Energy Principle"). It seems the loss-function for weight probability distributions is, in fact, the variational free energy. Hence, DKL[qψ(w|st+1)||qψ(w)] literally describes comparing weights before and after training.
Ted Tinker
CC BY-SA 4.0
2022-12-11T13:06:02.970Z
You can use mathjax on this site.
nbro
CC BY-SA 4.0
2022-12-12T05:33:50.260Z
Oh, thank you. I'll try editing the question.
Ted Tinker
<reinforcement-learning><machine-learning><probability-distribution><artificial-curiosity><bayesian-reinforcement-learning>
Calculating Curiosity with Friston's Free Energy in Reinforcement Learning
Ted Tinker
# Calculating Curiosity with Friston's Free Energy in Reinforcement Learning

(Asked by: username_0 on 2022-11-18)

I'm trying to implement the paper *[A Curiosity Algorithm for Robots Based on the Free Energy Principle](https://ieeexplore.ieee.org/document/9708819)* in a reinforcement learning environment using PyTorch, but I am unclear how curiosity is calculated.

As the paper describes, I made a 'transitioner' which predicts the next state given the current state and action. One layer of that transitioner is a Bayesian linear layer from [Blitz](https://github.com/piEsposito/blitz-bayesian-deep-learning); the probability distribution of weights in that layer is $q\_ψ(w) = N(w|μ,σ\_2)$ where $ψ = \{μ,σ\}$. Then, curiosity is defined as the Kullback–Leibler divergence $D\_{KL}[q\_ψ(w|s\_{t+1})||q\_ψ(w)]$.

$q\_ψ(w)$ is easily available, but $q\_ψ(w|s\_{t+1})$ is described as value of $q$ minimizing free energy $F = D\_{KL}[q\_ψ(w)||p\_ψ(w)] − E\_{q\_ψ}(w)[log (p\_ψ(s\_{t+1}|w))]$, and $p\_ψ$ isn't given. I think I understand the concept, but I'm puzzled how to implement this.

**Tags:** reinforcement-learning, machine-learning, probability-distribution, artificial-curiosity, bayesian-reinforcement-learning

> **Comments:**
> **username_0**: To update: I am learning much by focusing on the paper inspiring Blitz: "Weight Uncertainty in Neural Networks" (also cited in "A Curiosity Algorithm for Robots Based on the Free Energy Principle"). It seems the loss-function for weight probability distributions is, in fact, the variational free energy. Hence, DKL[qψ(w|st+1)||qψ(w)] literally describes comparing weights before and after training.  
> **username_1**: You can use mathjax on this site.  
> **username_0**: Oh, thank you. I'll try editing the question.  

---

## 0 answers

