<p>I'm making a simple deep Q learning algorithm, with cartpole-v1 env.</p>
<p>Like you can see in chart, after many episodes the reward decrease, some possible reasons?</p>
<p>The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)</p>
<p>The neural network is made from scratch
the complete code is here:
<a href="https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project" rel="nofollow noreferrer">https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project</a></p>
<pre><code># %%
from core.dqn_agent import DQNAgent
from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
from cartpole.cartpole_wrapper import CartPoleWrapper
import gym
import numpy as np
import torch
from tqdm import tqdm
import glob
import os
from IPython.display import Video
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from itertools import cycle
import sys
import shutil
from pathlib import Path
import shutil
import pyvirtualdisplay
_display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
_ = _display.start()



# %% [markdown]
# Initialize deep Q-learning agent, neural network, and parameters
# %%
np.random.seed(20)
agent = DQNAgent(env=CartPoleWrapper(gym.make(&quot;CartPole-v1&quot;)),
                nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)

DISCOUNT_FACTOR = 0.995
LEARNING_RATE = 0.0001

n_episodes = []
total_rewards = []
number_steps = []
total_episodes = 0


# %% [markdown]
# Training
# %%
while total_episodes &lt;= 10000:
    total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
    print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
    n_episodes.append(total_episodes)
    total_rewards.append(total_reward)
    number_steps.append(steps)

    for i in tqdm(range(50), 'learning...'):
        agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
    total_episodes += i+1

</code></pre>
<p><a href="https://i.sstatic.net/iT3h5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iT3h5.png" alt="chart" /></a></p>

CC BY-SA 4.0
2021-05-11T19:57:03.937Z
Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help.
S2673
CC BY-SA 4.0
2021-05-12T07:03:38.150Z
Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why
Lorenzo Tinfena
CC BY-SA 4.0
2021-05-27T00:09:40.317Z
Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case.
ddaedalus
CC BY-SA 4.0
2021-05-27T14:53:05.057Z
batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay, 
 or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint!
Lorenzo Tinfena
<machine-learning><reinforcement-learning><deep-learning><q-learning><dqn>
Reward firstly increase, but after more episodes, start decrease, and weights diverges
Lorenzo Tinfena
# Reward firstly increase, but after more episodes, start decrease, and weights diverges

(Asked by: username_0 on 2021-05-06)

I'm making a simple deep Q learning algorithm, with cartpole-v1 env.
Like you can see in chart, after many episodes the reward decrease, some possible reasons?
The exploration vs axplotation algorithm is epsilon-decay, I used a target network (used every mini-batch gradient descent update, in calculating actual Q values, and next Q values, is it right?)
The neural network is made from scratch
the complete code is here:
https://github.com/LorenzoTinfena/deep-q-learning-itt-final-project
# %%
from core.dqn_agent import DQNAgent
from cartpole.cartpole_neural_network import CartPoleNeuralNetwork
from cartpole.cartpole_wrapper import CartPoleWrapper
import gym
import numpy as np
import torch
from tqdm import tqdm
import glob
import os
from IPython.display import Video
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from itertools import cycle
import sys
import shutil
from pathlib import Path
import shutil
import pyvirtualdisplay
_display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
_ = _display.start()



# %% [markdown]
# Initialize deep Q-learning agent, neural network, and parameters
# %%
np.random.seed(20)
agent = DQNAgent(env=CartPoleWrapper(gym.make("CartPole-v1")),
                nn=CartPoleNeuralNetwork(), replay_memory_max_size=5000, batch_size=30)

DISCOUNT_FACTOR = 0.995
LEARNING_RATE = 0.0001

n_episodes = []
total_rewards = []
number_steps = []
total_episodes = 0


# %% [markdown]
# Training
# %%
while total_episodes <= 10000:
    total_reward, steps = agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=0, min_epsilon=0, render=False, optimize=False)
    print(f'\ntotal_episodes_training: {total_episodes}\tsteps: {steps}\ttotal_reward: {total_reward}', flush = True)
    n_episodes.append(total_episodes)
    total_rewards.append(total_reward)
    number_steps.append(steps)

    for i in tqdm(range(50), 'learning...'):
        agent.start_episode_and_evaluate(DISCOUNT_FACTOR, LEARNING_RATE, epsilon=1, epsilon_decay=0.99, min_epsilon=0.01, render=False, optimize=True)
    total_episodes += i+1

**Tags:** machine-learning, reinforcement-learning, deep-learning, q-learning, dqn

> **Comments:**
> **username_1**: Because it did well at the beginning, it could be that the simpler algorithm you’re using is just less consistent and updates that decrease the rewards sometimes happen. Decreasing the learning rate could help.  
> **username_0**: Thank you, now I improved algorithm, and at episode 320 the medium reward of 350, for me very well, but after episode 320 it start decrease algorithm, I heard about this fact, that after a maximum, start descrease, but I don't kow why  
> **username_2**: Try having a bigger batch size. There maybe some really bad batches that make your weights diverge. You can also use prioritized exp.replay ti check if this is the case.  
> **username_0**: batch size is 10 000, very big! yeah maybe in future I can try prioritized experience replay,   or maybe duel Q learning, I don't know, but for now I think is enough for a high school exam, thank ou for the hint!  

---

## 0 answers

