Answers.Body: <p>Sutton &amp; Barto are stating that if the initial and resultant policies have the same value function <strong>after policy improvement</strong>, then both are optimal because the Bellman optimality equation is satisfied.</p>
  <p>Section 3.6 discusses the following (true) statement: if two policies are optimal, then both have the same value function. The converse -- if two policies have the same value function, then both are optimal -- is <strong>not true</strong> for this statement. A simple counterexample can be constructed using a multi-armed bandit setup in which there are 3 actions: actions 1, 2, and 3 always yield rewards of 100, 100, and 200, respectively. The deterministic policies of choosing actions 1 and 2 have the same value function (a value of 100 at the start state, which is the only nonterminal state), but those policies are not optimal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-10T19:57:53.727Z
Answers.Text: Thanks for the response. I see through your example that two different policies can have the same value function and not be optimal. I have just one more question, in Bellman optimal equation, you already have the optimal value function in both the left and right side of the equation. But during policy improvement optimization, I think you don’t have an optimal function in neither side. Well, At least while the sides are not equal and the process has not converged. I suppose that a proof of policy improvement should show that equality implies that value is optimal.
Answers.UserDisplayName: User1983
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-05T22:15:12.713Z
Answers.LastActivityDate: 2024-10-05T22:15:12.713Z
Answers.UserDisplayName: DeepQZero
Body: <p>According <a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton and Barto</a> second edition, page 79, policy improvement must give a better policy except when the policy is already optimal. This means that if two policies have the same value function then both are optimal. In other words, you cannot have two different non optimal policies with the same value function. Is this correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-05T22:23:40.260Z
Comments.Text: Hello @User1983 and welcome to AI Stack Exchange! Thank you for clearly defining your question in the title and body of the post. We hope to see more of your questions in the near future!
Comments.UserDisplayName: DeepQZero
Tags: <reinforcement-learning><machine-learning><deep-learning><deep-rl>
Title: Can two different non-optimal policies have the same value functions?
UserDisplayName: User1983
ThreadText: # Can two different non-optimal policies have the same value functions?
  
  (Asked by: username_0 on 2024-10-05)
  
  According [Sutton and Barto](http://incompleteideas.net/book/the-book-2nd.html) second edition, page 79, policy improvement must give a better policy except when the policy is already optimal. This means that if two policies have the same value function then both are optimal. In other words, you cannot have two different non optimal policies with the same value function. Is this correct?
  
  **Tags:** reinforcement-learning, machine-learning, deep-learning, deep-rl
  > **Comments:**
  > **username_1**: Hello @User1983 and welcome to AI Stack Exchange! Thank you for clearly defining your question in the title and body of the post. We hope to see more of your questions in the near future!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-05. Score: 7 | Accepted answer
  
  Sutton & Barto are stating that if the initial and resultant policies have the same value function **after policy improvement**, then both are optimal because the Bellman optimality equation is satisfied.
  
  Section 3.6 discusses the following (true) statement: if two policies are optimal, then both have the same value function. The converse -- if two policies have the same value function, then both are optimal -- is **not true** for this statement. A simple counterexample can be constructed using a multi-armed bandit setup in which there are 3 actions: actions 1, 2, and 3 always yield rewards of 100, 100, and 200, respectively. The deterministic policies of choosing actions 1 and 2 have the same value function (a value of 100 at the start state, which is the only nonterminal state), but those policies are not optimal.
  
  > **Comments:**
  > **username_0**: Thanks for the response. I see through your example that two different policies can have the same value function and not be optimal. I have just one more question, in Bellman optimal equation, you already have the optimal value function in both the left and right side of the equation. But during policy improvement optimization, I think you don’t have an optimal function in neither side. Well, At least while the sides are not equal and the process has not converged. I suppose that a proof of policy improvement should show that equality implies that value is optimal.  
  
  ---
  
  