Body: <p>I'm making a simple deep Q learning algorithm, with cartpole-v1 env.</p>
  <p>Like you can see in chart, after many episodes the reward decrease, some possible reasons?</p>
  <p>The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)</p>
  <p>The neural network is made from scratch
  the complete code is here:
  <a href="https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project" rel="nofollow noreferrer">https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project</a></p>
  <pre><code># %%
  from core.dqn_agent import DQNAgent
  from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
  from cartpole.cartpole_wrapper import CartPoleWrapper
  import gym
  import numpy as np
  import torch
  from tqdm import tqdm
  import glob
  import os
  from IPython.display import Video
  import matplotlib
  import matplotlib.pyplot as plt
  import numpy as np
  from itertools import cycle
  import sys
  import shutil
  from pathlib import Path
  import shutil
  import pyvirtualdisplay
  _display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
  _ = _display.start()
  
  
  
  # %% [markdown]
  # Initialize deep Q-learning agent, neural network, and parameters
  # %%
  np.random.seed(20)
  agent = DQNAgent(env=CartPoleWrapper(gym.make(&quot;CartPole-v1&quot;)),
                  nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)
  
  DISCOUNT_FACTOR = 0.995
  LEARNING_RATE = 0.0001
  
  n_episodes = []
  total_rewards = []
  number_steps = []
  total_episodes = 0
  
  
  # %% [markdown]
  # Training
  # %%
  while total_episodes &lt;= 10000:
      total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
      print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
      n_episodes.append(total_episodes)
      total_rewards.append(total_reward)
      number_steps.append(steps)
  
      for i in tqdm(range(50), 'learning...'):
          agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
      total_episodes += i+1
  
  </code></pre>
  <p><a href="https://i.sstatic.net/iT3h5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iT3h5.png" alt="chart" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-11T19:57:03.937Z
Comments.Text: Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-12T07:03:38.150Z
Comments.Text: Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why
Comments.UserDisplayName: Lorenzo Tinfena
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-27T00:09:40.317Z
Comments.Text: Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case.
Comments.UserDisplayName: ddaedalus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-27T14:53:05.057Z
Comments.Text: batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay, 
   or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint!
Comments.UserDisplayName: Lorenzo Tinfena
Tags: <machine-learning><reinforcement-learning><deep-learning><q-learning><dqn>
Title: Reward firstly increase, but after more episodes, start decrease, and weights diverges
UserDisplayName: Lorenzo Tinfena
ThreadText: # Reward firstly increase, but after more episodes, start decrease, and weights diverges
  
  (Asked by: username_0 on 2021-05-06)
  
  I'm making a simple deep Q learning algorithm, with cartpole-v1 env.
  Like you can see in chart, after many episodes the reward decrease, some possible reasons?
  The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)
  The neural network is made from scratch
  the complete code is here:
  https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project
  # %%
  from core.dqn_agent import DQNAgent
  from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
  from cartpole.cartpole_wrapper import CartPoleWrapper
  import gym
  import numpy as np
  import torch
  from tqdm import tqdm
  import glob
  import os
  from IPython.display import Video
  import matplotlib
  import matplotlib.pyplot as plt
  import numpy as np
  from itertools import cycle
  import sys
  import shutil
  from pathlib import Path
  import shutil
  import pyvirtualdisplay
  _display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
  _ = _display.start()
  
  
  
  # %% [markdown]
  # Initialize deep Q-learning agent, neural network, and parameters
  # %%
  np.random.seed(20)
  agent = DQNAgent(env=CartPoleWrapper(gym.make("CartPole-v1")),
                  nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)
  
  DISCOUNT_FACTOR = 0.995
  LEARNING_RATE = 0.0001
  
  n_episodes = []
  total_rewards = []
  number_steps = []
  total_episodes = 0
  
  
  # %% [markdown]
  # Training
  # %%
  while total_episodes <= 10000:
      total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
      print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
      n_episodes.append(total_episodes)
      total_rewards.append(total_reward)
      number_steps.append(steps)
  
      for i in tqdm(range(50), 'learning...'):
          agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
      total_episodes += i+1
  
  **Tags:** machine-learning, reinforcement-learning, deep-learning, q-learning, dqn
  > **Comments:**
  > **username_1**: Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help.  
  > **username_0**: Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why  
  > **username_2**: Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case.  
  > **username_0**: batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay,   or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint!  
  
  ---
  
  ## 0 answers
  
  