Body: <p>I have implemented a simple version of the DQN algorithm for <code>CartPole-v0</code>. The algorithm works fine, in the sense that achieves the highest possible scores. The below diagram shows the cumulative reward versus training episode.
  <a href="https://i.sstatic.net/F7VTa.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/F7VTa.png" alt="reward plot" /></a></p>
  <p>The scary part is when I tried to plot the q values during training. For this purpose, 1000 random states were generated and stored. After each training episode, I fed these states to the Q-network and computed the Q-value for all actions. Then for each state, the max Q-value was computed. Finally, these max Q-values were averaged to yield a single number. The following plot shows this quantity during training.
  <a href="https://i.sstatic.net/yr91j.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yr91j.png" alt="enter image description here" /></a></p>
  <p>As you can see, there is a slight increase in average-max-q in the beginning and it is followed by a sharp decrease. My question is, <strong>how can this value be negative?</strong> Since I know all rewards received by the agent are positive, and expected cumulative reward of a state can not be negative.</p>
  <h1>Edit</h1>
  <p>I added the code for clarity:</p>
  <pre><code>from matplotlib import pyplot as plt
  
  import torch as th
  import torch.nn as nn
  import torch.nn.functional as F
  
  import gym
  import random
  import numpy as np
  from collections import deque
  
  from test_avg_q import TestEnv
  
  
  class ReplayBuffer():
      def __init__(self, maxlen):
          self.buffer = deque(maxlen=maxlen)
  
      def add(self, experience):
          self.buffer.append(experience)
  
      def sample(self, batch_size):
          sample_size = min(len(self.buffer), batch_size)
          samples = random.choices(self.buffer, k=sample_size)
          return map(list, zip(*samples))
  
  
  class QNetwork():
      def __init__(self, state_dim, action_size):
          self.action_size = action_size
          self.q_net = nn.Sequential(nn.Linear(state_dim, 100),
                                     nn.ReLU(),
                                     nn.Linear(100, action_size))
          self.optimizer = th.optim.Adam(self.q_net.parameters(), lr=0.001)
  
      def update_model(self, state, action, q_target):
          action = th.Tensor(action).to(th.int64)
          action_one_hot = F.one_hot(action, num_classes=self.action_size)
          q = self.q_net(th.Tensor(state))
          q_a = th.sum(q * action_one_hot, dim=1)
          loss = nn.MSELoss()(q_a, q_target)
          self.optimizer.zero_grad()
          loss.backward()
          self.optimizer.step()
  
  
  class DQNAgent():
      def __init__(self, env):
          self.state_dim = env.observation_space.shape[0]
          self.action_size = env.action_space.n
          self.q_network = QNetwork(self.state_dim, self.action_size)
          self.replay_buffer = ReplayBuffer(maxlen=1_000_000)
          self.gamma = 0.97
          self.eps = 1.0
  
      def get_action(self, state):
          with th.no_grad():
              q_state = self.q_network.q_net(th.Tensor(state).unsqueeze(0))
              action_greedy = th.argmax(q_state).item()
          action_random = np.random.randint(self.action_size)
          action = action_random if random.random() &lt; self.eps else action_greedy
          return action
  
      def train(self, state, action, next_state, reward, done):
          self.replay_buffer.add((state, action, next_state, reward, done))
          states, actions, next_states, rewards, dones = self.replay_buffer.sample(50)
          with th.no_grad():
              q_next_states = self.q_network.q_net(th.Tensor(next_states))
              q_next_states[dones] = th.zeros(self.action_size)
              q_targets = th.Tensor(rewards) + self.gamma * th.max(q_next_states, dim=1)[0]
          self.q_network.update_model(states, actions, q_targets)
  
          if done: self.eps = max(0.1, 0.99 * self.eps)
  
  
  env_name = &quot;CartPole-v0&quot;
  env = gym.make(env_name)
  
  agent = DQNAgent(env)
  num_episodes = 400
  testEnv = TestEnv()
  avg_qs = []
  rewards = []
  render = False
  
  for ep in range(num_episodes):
      state = env.reset()
      total_reward = 0
      done = False
      while not done:
          action = agent.get_action(state)
          next_state, reward, done, info = env.step(action)
          agent.train(state, action, next_state, reward, done)
          total_reward += reward
          state = next_state
          if render and ep &gt; 80:
              env.render()
  
      avg_qs.append(testEnv.run(agent.q_network.q_net))
      rewards.append(total_reward)
      print(&quot;Episode: {}, total_reward: {:.2f}&quot;.format(ep, total_reward))
  
  plt.plot(avg_qs)
  plt.show()
  plt.figure()
  plt.plot(rewards)
  plt.show()
  
  </code></pre>
  <p>here is the code for class <code>TestEnv</code>:</p>
  <pre><code>import torch as th
  from torch.utils.data import Dataset, DataLoader
  
  
  class TestEnv:
      def __init__(self):
          self.dataloader = DataLoader(TestDataset(), batch_size=100, num_workers=10)
  
      def run(self, model):
          out_list = []
          model.eval()
          with th.no_grad():
              for batch in self.dataloader:
                  out = model(batch)
                  out_list.append(out)
              qs = th.cat(out_list, dim=0)
              maxq = th.max(qs, dim=1)[0]
              result = th.mean(maxq)
          return result.item()
  
  
  
  class TestDataset(Dataset):
      def __init__(self):
          self.db = th.load('test_states.pt')
  
      def __len__(self):
          return len(self.db)
  
      def __getitem__(self, idx):
          return self.db[idx]
  
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-08T09:31:39.853Z
Comments.Text: Maybe link your implementation. I would suspect a bug in your code somewhere, as you point out accurate Q values for real states should at least be positive. I don't think it is possible to predict where the bug is from the graphs.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-10T10:49:45.510Z
Comments.Text: for me, it would be possible that your 1000 random states are not a good representation of the gigantic state space of the problem, most states in the problem yield very bad outcome, only a part of them can be considered of interest (e.g. the ones where the pole is more of less straight).
  if your 1000 all lead to certain fail, the max expected reward for those states can only go down, because you never want to reach them, nor their child.
Comments.UserDisplayName: JeanMi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-04T00:09:02.673Z
Comments.Text: I'm having a similar problem (mean q-value increases and then decreases). Did you figure this out?
Comments.UserDisplayName: Luis Alejandro Snchez Prez PhD
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-06T01:42:24.730Z
Comments.Text: Look up the deadly triad. There is a lot of literature devoted to the overestimation of Q values.
Comments.UserDisplayName: desert_ranger
Tags: <deep-rl><dqn><gym>
Title: Why does Q-value become negative during training of DQN, while the agent learns to play?
UserDisplayName: Arman
ThreadText: # Why does Q-value become negative during training of DQN, while the agent learns to play?
  
  (Asked by: username_0 on 2021-03-08)
  
  I have implemented a simple version of the DQN algorithm for CartPole-v0. The algorithm works fine, in the sense that achieves the highest possible scores. The below diagram shows the cumulative reward versus training episode.
  
  The scary part is when I tried to plot the q values during training. For this purpose, 1000 random states were generated and stored. After each training episode, I fed these states to the Q-network and computed the Q-value for all actions. Then for each state, the max Q-value was computed. Finally, these max Q-values were averaged to yield a single number. The following plot shows this quantity during training.
  
  As you can see, there is a slight increase in average-max-q in the beginning and it is followed by a sharp decrease. My question is, how can this value be negative? Since I know all rewards received by the agent are positive, and expected cumulative reward of a state can not be negative.
  Edit
  I added the code for clarity:
  from matplotlib import pyplot as plt
  
  import torch as th
  import torch.nn as nn
  import torch.nn.functional as F
  
  import gym
  import random
  import numpy as np
  from collections import deque
  
  from test_avg_q import TestEnv
  
  
  class ReplayBuffer():
      def __init__(self, maxlen):
          self.buffer = deque(maxlen=maxlen)
  
      def add(self, experience):
          self.buffer.append(experience)
  
      def sample(self, batch_size):
          sample_size = min(len(self.buffer), batch_size)
          samples = random.choices(self.buffer, k=sample_size)
          return map(list, zip(*samples))
  
  
  class QNetwork():
      def __init__(self, state_dim, action_size):
          self.action_size = action_size
          self.q_net = nn.Sequential(nn.Linear(state_dim, 100),
                                     nn.ReLU(),
                                     nn.Linear(100, action_size))
          self.optimizer = th.optim.Adam(self.q_net.parameters(), lr=0.001)
  
      def update_model(self, state, action, q_target):
          action = th.Tensor(action).to(th.int64)
          action_one_hot = F.one_hot(action, num_classes=self.action_size)
          q = self.q_net(th.Tensor(state))
          q_a = th.sum(q * action_one_hot, dim=1)
          loss = nn.MSELoss()(q_a, q_target)
          self.optimizer.zero_grad()
          loss.backward()
          self.optimizer.step()
  
  
  class DQNAgent():
      def __init__(self, env):
          self.state_dim = env.observation_space.shape[0]
          self.action_size = env.action_space.n
          self.q_network = QNetwork(self.state_dim, self.action_size)
          self.replay_buffer = ReplayBuffer(maxlen=1_000_000)
          self.gamma = 0.97
          self.eps = 1.0
  
      def get_action(self, state):
          with th.no_grad():
              q_state = self.q_network.q_net(th.Tensor(state).unsqueeze(0))
              action_greedy = th.argmax(q_state).item()
          action_random = np.random.randint(self.action_size)
          action = action_random if random.random() < self.eps else action_greedy
          return action
  
      def train(self, state, action, next_state, reward, done):
          self.replay_buffer.add((state, action, next_state, reward, done))
          states, actions, next_states, rewards, dones = self.replay_buffer.sample(50)
          with th.no_grad():
              q_next_states = self.q_network.q_net(th.Tensor(next_states))
              q_next_states[dones] = th.zeros(self.action_size)
              q_targets = th.Tensor(rewards) + self.gamma * th.max(q_next_states, dim=1)[0]
          self.q_network.update_model(states, actions, q_targets)
  
          if done: self.eps = max(0.1, 0.99 * self.eps)
  
  
  env_name = "CartPole-v0"
  env = gym.make(env_name)
  
  agent = DQNAgent(env)
  num_episodes = 400
  testEnv = TestEnv()
  avg_qs = []
  rewards = []
  render = False
  
  for ep in range(num_episodes):
      state = env.reset()
      total_reward = 0
      done = False
      while not done:
          action = agent.get_action(state)
          next_state, reward, done, info = env.step(action)
          agent.train(state, action, next_state, reward, done)
          total_reward += reward
          state = next_state
          if render and ep > 80:
              env.render()
  
      avg_qs.append(testEnv.run(agent.q_network.q_net))
      rewards.append(total_reward)
      print("Episode: {}, total_reward: {:.2f}".format(ep, total_reward))
  
  plt.plot(avg_qs)
  plt.show()
  plt.figure()
  plt.plot(rewards)
  plt.show()
  
  
  here is the code for class TestEnv:
  import torch as th
  from torch.utils.data import Dataset, DataLoader
  
  
  class TestEnv:
      def __init__(self):
          self.dataloader = DataLoader(TestDataset(), batch_size=100, num_workers=10)
  
      def run(self, model):
          out_list = []
          model.eval()
          with th.no_grad():
              for batch in self.dataloader:
                  out = model(batch)
                  out_list.append(out)
              qs = th.cat(out_list, dim=0)
              maxq = th.max(qs, dim=1)[0]
              result = th.mean(maxq)
          return result.item()
  
  
  
  class TestDataset(Dataset):
      def __init__(self):
          self.db = th.load('test_states.pt')
  
      def __len__(self):
          return len(self.db)
  
      def __getitem__(self, idx):
          return self.db[idx]
  
  **Tags:** deep-rl, dqn, gym
  > **Comments:**
  > **username_1**: Maybe link your implementation. I would suspect a bug in your code somewhere, as you point out accurate Q values for real states should at least be positive. I don't think it is possible to predict where the bug is from the graphs.  
  > **username_2**: for me, it would be possible that your 1000 random states are not a good representation of the gigantic state space of the problem, most states in the problem yield very bad outcome, only a part of them can be considered of interest (e.g. the ones where the pole is more of less straight). if your 1000 all lead to certain fail, the max expected reward for those states can only go down, because you never want to reach them, nor their child.  
  > **username_3**: I'm having a similar problem (mean q-value increases and then decreases). Did you figure this out?  
  > **username_4**: Look up the deadly triad. There is a lot of literature devoted to the overestimation of Q values.  
  
  ---
  
  ## 0 answers
  
  