Body: <p>Here is the link to the paper <a href="https://www.davidsilver.uk/wp-content/uploads/2020/03/mc_aixi_long.pdf" rel="nofollow noreferrer">https://www.davidsilver.uk/wp-content/uploads/2020/03/mc_aixi_long.pdf</a></p>
  <p>Definition 2. An environment <span class="math-container">$\rho$</span> is a sequence of conditional probability functions <span class="math-container">$\{ \rho_0, \rho_1, \rho_2, ...\}$</span>, where <span class="math-container">$\rho_n: A^n \rightarrow Density(X^n)$</span> that satisfies</p>
  <p><span class="math-container">$$ \forall a_{i:n} \forall x_{&lt;n} : \rho_{n-1}(x_{&lt;n}|a_{&lt;n}) = \sum_{x_n \in X} \rho_n (x_{1:n} | a_{1:n})$$</span></p>
  <p>In the base case, we have <span class="math-container">$\rho_0(\epsilon | \epsilon) = 1$</span></p>
  <p>Ok so the right hand side seems to be saying, that <span class="math-container">$x_{1:n}$</span> will be made from <span class="math-container">$x_{&lt;n}$</span> concatenated with <span class="math-container">$x_n \in X$</span>, and the sum will be over these different realizations of <span class="math-container">$x_{1:n}$</span>, so the probability function for the shorter sequence <span class="math-container">$\rho_{n-1}(x_{&lt;n} | a_{&lt;n})$</span> depends on the possible longer sequences? This seems to be what the equation is saying but it doesn't make sense that the present probability of a sequence depends on possible future outcomes.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-04T23:26:33.187Z
Comments.Text: I'm not familiar with this paper, although it's one I've been trying to dedicate some time to read, but, if you read their explanation of how we should intuitively look at the AIXI decision rule, it shouldn't be surprising that something in the present depends on the future (aka planning). The idea that an environment is a sequence of probability distributions that map actions to rewards and observations is fine. But currently that equation is also not clear to me. It looks a bit like the law of total probability.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-04T23:26:48.090Z
Comments.Text: They call it the "chronological condition" and refer to Hutter 2005. I tried to read that book a few years ago. It's very technical so not easy to read, but quite interesting
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><probability><aixi>
Title: Trying to understand the definition of environment in this paper on monte carlo approximation of AIXI
UserDisplayName: TomT800
ThreadText: # Trying to understand the definition of environment in this paper on monte carlo approximation of AIXI
  
  (Asked by: username_0 on 2023-06-02)
  
  Here is the link to the paper <https://www.davidsilver.uk/wp-content/uploads/2020/03/mc_aixi_long.pdf>
  
  Definition 2. An environment $\rho$ is a sequence of conditional probability functions $\{ \rho\_0, \rho\_1, \rho\_2, ...\}$, where $\rho\_n: A^n \rightarrow Density(X^n)$ that satisfies
  
  $$ \forall a\_{i:n} \forall x\_{
  
  In the base case, we have $\rho\_0(\epsilon | \epsilon) = 1$
  
  Ok so the right hand side seems to be saying, that $x\_{1:n}$ will be made from $x\_{ concatenated with $x\_n \in X$, and the sum will be over these different realizations of $x\_{1:n}$, so the probability function for the shorter sequence $\rho\_{n-1}(x\_{ depends on the possible longer sequences? This seems to be what the equation is saying but it doesn't make sense that the present probability of a sequence depends on possible future outcomes.
  
  **Tags:** reinforcement-learning, probability, aixi
  > **Comments:**
  > **username_1**: I'm not familiar with this paper, although it's one I've been trying to dedicate some time to read, but, if you read their explanation of how we should intuitively look at the AIXI decision rule, it shouldn't be surprising that something in the present depends on the future (aka planning). The idea that an environment is a sequence of probability distributions that map actions to rewards and observations is fine. But currently that equation is also not clear to me. It looks a bit like the law of total probability.  
  > **username_1**: They call it the "chronological condition" and refer to Hutter 2005. I tried to read that book a few years ago. It's very technical so not easy to read, but quite interesting  
  
  ---
  
  ## 0 answers
  
  