Answers.Body: <p>Unlike a game with a fixed target state, determining when the SARSA algorithm for an inverted pendulum problem has converged can be tricky. You may try below heuristic convergence criterion or combining them dynamically for your problem.</p>
  <p>Since you know for the policy to be considered good the expected reward should be -200 or better, you can leverage cumulative reward per episode as a convergence criterion. If the cumulative reward remains consistently above -200 for a number of episodes (e.g., 100), you can assume the policy has converged.</p>
  <p>Or you can monitor the change in the Q-values during training. After each Q-value update, check how much the Q-values change. If the Q-values stop changing significantly (e.g., with an absolute threshold of 0.01 or percentage threshold 1%) over multiple episodes (e.g., 100), it suggests that the policy has converged to a stable one.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-02T07:23:42.407Z
Answers.LastActivityDate: 2024-12-02T07:23:42.407Z
Answers.UserDisplayName: cinch
Body: <p>I have an Inverted pendulum domain, where I need to discretize state space (defined by 2 state variables - angle and angular velocity with given ranges) and action space (torque values with given bounds) and implement a SARSA algorithm, to arrive at optimal q-value and optimal policy. Each episode terminates after a fixed no. of time steps say T (and is not defined by any specific terminal state).</p>
  <p>I'm confused on how to decide when the algorithm has converged?</p>
  <p>Let's say I use a epsilon greedy action, with some epsilon say 0.1, and learning rate say 0.05 to do the TD update in each iteration, and say a discount factor of 1. Now of course I have to tweak these hyperparameters (epsilon, alpha (learning rate), q-value initialization (random/zero/optimistic, etc) to figure out the best performing values.  There is an equation defined to compute reward - as a function of angle, angular velocity in each time step (which I believe can be abstracted to answer this question). First, I start by discretizing the states and actions to random no. of bin, say 1 bin, for each unit difference. (for eg. torque --&gt; -2-&gt;(-1),(-1)-&gt;0, 0-&gt;1, 1-&gt;2, etc)</p>
  <p>I don't know what the optimal value function is, however, I know for the policy to be considered good, the expected reward -200 or better. How to use this to arrive at a convergence constraint for my SARSA implementation.</p>
  
Tags: <reinforcement-learning><temporal-difference-methods><sarsa>
Title: SARSA Implementation - how to arrive at a convergence constraint, if the optimal value function is not known?
UserDisplayName: jarvis
ThreadText: # SARSA Implementation - how to arrive at a convergence constraint, if the optimal value function is not known?
  
  (Asked by: username_0 on 2024-12-01)
  
  I have an Inverted pendulum domain, where I need to discretize state space (defined by 2 state variables - angle and angular velocity with given ranges) and action space (torque values with given bounds) and implement a SARSA algorithm, to arrive at optimal q-value and optimal policy. Each episode terminates after a fixed no. of time steps say T (and is not defined by any specific terminal state).
  
  I'm confused on how to decide when the algorithm has converged?
  
  Let's say I use a epsilon greedy action, with some epsilon say 0.1, and learning rate say 0.05 to do the TD update in each iteration, and say a discount factor of 1. Now of course I have to tweak these hyperparameters (epsilon, alpha (learning rate), q-value initialization (random/zero/optimistic, etc) to figure out the best performing values. There is an equation defined to compute reward - as a function of angle, angular velocity in each time step (which I believe can be abstracted to answer this question). First, I start by discretizing the states and actions to random no. of bin, say 1 bin, for each unit difference. (for eg. torque --> -2->(-1),(-1)->0, 0->1, 1->2, etc)
  
  I don't know what the optimal value function is, however, I know for the policy to be considered good, the expected reward -200 or better. How to use this to arrive at a convergence constraint for my SARSA implementation.
  
  **Tags:** reinforcement-learning, temporal-difference-methods, sarsa
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-02. Score: 0
  
  Unlike a game with a fixed target state, determining when the SARSA algorithm for an inverted pendulum problem has converged can be tricky. You may try below heuristic convergence criterion or combining them dynamically for your problem.
  
  Since you know for the policy to be considered good the expected reward should be -200 or better, you can leverage cumulative reward per episode as a convergence criterion. If the cumulative reward remains consistently above -200 for a number of episodes (e.g., 100), you can assume the policy has converged.
  
  Or you can monitor the change in the Q-values during training. After each Q-value update, check how much the Q-values change. If the Q-values stop changing significantly (e.g., with an absolute threshold of 0.01 or percentage threshold 1%) over multiple episodes (e.g., 100), it suggests that the policy has converged to a stable one.
  
  
  ---
  
  