Answers.Body: <p>I figured this out by going to the author's publicly available github code. It turned out  the authors were just generating the transition probability <span class="math-container">$p$</span> from <span class="math-container">$\mathcal{N}(\mu,\sigma^2)$</span> at the beginning of each episode for some reason. Answering it myself for the sake of not leaving this question unanswered.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-08T00:34:37.617Z
Answers.Text: Wait, but $p$ needs to be limited to $[0,1]$, if it's a probability. However, the Gaussian is not limited to $[0, 1]$. Maybe you could also provide a link to the source code.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-08T00:51:37.637Z
Answers.Text: @nbro I misworded it - the maximum number of episodes is $T = 20$ and the balloon explodes if the current the number of steps have taken so far $t$ exceeds the normal random variable drawn from $\mathcal{N}(16, 0.5),$ for the low-risk version of the task and $\mathcal{N}(8, 0.5)$ for the high-risk version.
Answers.UserDisplayName: dezdichado
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-08T00:14:51.430Z
Answers.LastActivityDate: 2021-02-08T00:14:51.430Z
Answers.UserDisplayName: dezdichado
Body: <p>I am reading this paper <a href="https://www.mitpressjournals.org/doi/full/10.1162/cpsy_a_00026" rel="nofollow noreferrer">Anxiety, Avoidance and Sequential Evaluation</a> and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a simple MDP for which the description is below:</p>
  <p><a href="https://i.sstatic.net/jZHwa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jZHwa.png" alt="enter image description here" /></a></p>
  <p>My confusion is the following sentence:</p>
  <blockquote>
  <p>...The probability of this bad transition was modeled using normal density function, with parameters <span class="math-container">$N(16, 0.5)$</span></p>
  </blockquote>
  <p>But the fact that this is a continuous, normal distribution makes me stumped. In MDP's, usually there is a nice, discrete transition matrix and so there is no ambiguity as to how to implement it. For instance, if they said the transition to a bad state is modeled by a Bernoulli random variable with parameter <span class="math-container">$p,$</span> then it is clear how to implement it. I would do something like:</p>
  <pre><code>def step(curr_state, curr_action):
     if uniform random variable(0,1) &lt; p:
        next_state = bad state
  </code></pre>
  <p>But they are using a normal random variable for this &quot;bad&quot; transition, so how do I implement this?</p>
  
Tags: <reinforcement-learning><markov-decision-process><implementation><temporal-difference-methods><transition-model>
Title: How should I implement the state transition when it is a Gaussian distribution?
UserDisplayName: dezdichado
ThreadText: # How should I implement the state transition when it is a Gaussian distribution?
  
  (Asked by: username_0 on 2021-01-25)
  
  I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a simple MDP for which the description is below:
  
  My confusion is the following sentence:
  
  ...The probability of this bad transition was modeled using normal density function, with parameters $N(16, 0.5)$
  
  But the fact that this is a continuous, normal distribution makes me stumped. In MDP's, usually there is a nice, discrete transition matrix and so there is no ambiguity as to how to implement it. For instance, if they said the transition to a bad state is modeled by a Bernoulli random variable with parameter $p,$ then it is clear how to implement it. I would do something like:
  def step(curr_state, curr_action):
     if uniform random variable(0,1) < p:
        next_state = bad state
  
  But they are using a normal random variable for this "bad" transition, so how do I implement this?
  
  **Tags:** reinforcement-learning, markov-decision-process, implementation, temporal-difference-methods, transition-model
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-02-08. Score: 0 | Accepted answer
  
  I figured this out by going to the author's publicly available github code. It turned out the authors were just generating the transition probability $p$ from $\mathcal{N}(\mu,\sigma^2)$ at the beginning of each episode for some reason. Answering it myself for the sake of not leaving this question unanswered.
  
  > **Comments:**
  > **username_1**: Wait, but $p$ needs to be limited to $[0,1]$, if it's a probability. However, the Gaussian is not limited to $[0, 1]$. Maybe you could also provide a link to the source code.  
  > **username_0**: @nbro I misworded it - the maximum number of episodes is $T = 20$ and the balloon explodes if the current the number of steps have taken so far $t$ exceeds the normal random variable drawn from $\mathcal{N}(16, 0.5),$ for the low-risk version of the task and $\mathcal{N}(8, 0.5)$ for the high-risk version.  
  
  ---
  
  