Answers.Body: <blockquote>
  <p>At line 10: The value function is represented as V[s], which has bracket notation-like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?</p>
  </blockquote>
  <p>This is just a notation that the value function is a mapping between <em>S</em> and the real numbers. When implementing, you would want to store <em>V(s)</em> and <em>π(s)</em> as either <strong>arrays</strong> or some kind of <strong>hashmap</strong> like <code>unordered_map</code> (in which case your states must be hashable). Here we also have to assume that this container will fit in the memory, otherwise, the value function has to be approximated with function approximation which is not covered explicitly by this pseudo-code.</p>
  <blockquote>
  <p>At line 12: Vk would be the element in index k inside of array V?</p>
  </blockquote>
  <p>No. This line says that you would need to store multiple value functions during the algorithm, basically, a list of functions (or think of it as an array of functions or a two-dimensional array, where <em>k</em> goes along one dimension and <em>s</em> goes along the other dimension) and if you look at the loop following it, it shows you how you need to keep appending new value functions to the end of this list. However, notice (in line 19) that you only need the value function from the previous iteration to calculate your new value function, which means that you will never need to store more than two value functions (the new one and the previous one).</p>
  <blockquote>
  <p>At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.</p>
  </blockquote>
  <p>Yes. This loop calculates your value function.</p>
  <blockquote>
  <p>Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?</p>
  </blockquote>
  <p>Here you have to iterate through all the actions <em>a</em> in state <em>s</em> and calculate the sum according to the formula in the max function. Then you take the <strong>highest value</strong> and set the new <em>V(s)</em> to be this value.</p>
  <blockquote>
  <p>Line 20: I'm interpreting this a the end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So would I have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)</p>
  </blockquote>
  <p>You have to check for <strong>all</strong> <em>s</em> if the condition satisfies and you may only break the loop if all these values are lower than the threshold <em>θ</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-18T08:48:53.563Z
Answers.LastActivityDate: 2021-10-02T18:59:21.300Z
Answers.LastEditDate: 2021-10-02T18:59:21.300Z
Answers.UserDisplayName: Hai Nguyen
Body: <p>I am having a difficult time translating this pseudocode into functional C++ code.</p>
  <p><a href="https://i.sstatic.net/aJjyc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aJjyc.png" alt="enter image" /></a></p>
  <ul>
  <li>At line 10: The value function is represented as V[s], which has bracket notation-like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?</li>
  <li>At line 12: Vk would be the element in index k inside of array V?</li>
  <li>At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.</li>
  <li>Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?</li>
  <li>Line 20: I'm interpreting this as the the-end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So Would I would have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-18T02:48:24.140Z
Comments.Text: Can you add the code as text?  The file link (from my side) is broken.
Comments.UserDisplayName: solarflare
Tags: <reinforcement-learning><implementation><value-iteration><c++>
Title: Value iteration algorithm from pseudo-code to C++
UserDisplayName: ALostBegginer
ThreadText: # Value iteration algorithm from pseudo-code to C++
  
  (Asked by: username_0 on 2018-04-17)
  
  I am having a difficult time translating this pseudocode into functional C++ code.
  
  [![enter image](https://i.sstatic.net/aJjyc.png)](https://i.sstatic.net/aJjyc.png)
  
  * At line 10: The value function is represented as V[s], which has bracket notation-like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?
  * At line 12: Vk would be the element in index k inside of array V?
  * At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.
  * Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?
  * Line 20: I'm interpreting this as the the-end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So Would I would have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)
  
  **Tags:** reinforcement-learning, implementation, value-iteration, c++
  > **Comments:**
  > **username_1**: Can you add the code as text?  The file link (from my side) is broken.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-04-18. Score: 0 | Accepted answer
  
  > At line 10: The value function is represented as V[s], which has bracket notation-like arrays. Is this a separate method or just a function of the value with a given state? Why is S inside the brackets? Is this supposed to be an array with as many elements as S?
  
  This is just a notation that the value function is a mapping between *S* and the real numbers. When implementing, you would want to store *V(s)* and *π(s)* as either **arrays** or some kind of **hashmap** like `unordered_map` (in which case your states must be hashable). Here we also have to assume that this container will fit in the memory, otherwise, the value function has to be approximated with function approximation which is not covered explicitly by this pseudo-code.
  
  > At line 12: Vk would be the element in index k inside of array V?
  
  No. This line says that you would need to store multiple value functions during the algorithm, basically, a list of functions (or think of it as an array of functions or a two-dimensional array, where *k* goes along one dimension and *s* goes along the other dimension) and if you look at the loop following it, it shows you how you need to keep appending new value functions to the end of this list. However, notice (in line 19) that you only need the value function from the previous iteration to calculate your new value function, which means that you will never need to store more than two value functions (the new one and the previous one).
  
  > At line 16: I'm interpreting this as the start of a do-while loop that ends at line 20.
  
  Yes. This loop calculates your value function.
  
  > Line 19: I'm finding the action that maximizes the sum, for all states, of the equation following the sigma?
  
  Here you have to iterate through all the actions *a* in state *s* and calculate the sum according to the formula in the max function. Then you take the **highest value** and set the new *V(s)* to be this value.
  
  > Line 20: I'm interpreting this a the end of the do-while. But what is this condition? Am I checking if there is an s such that this condition applies? So would I have to loop between all states and stop if any state satisfies the condition? (Basically a loop with a break, instead of a while)
  
  You have to check for **all** *s* if the condition satisfies and you may only break the loop if all these values are lower than the threshold *θ*.
  
  
  ---
  
  