Sudden drop of score in the last few episodes
I was following this tutorial about lunar lander and deep Q learning with Tensorflow 2 and I noticed something odd.
The problem was actually solved at episode 476 but then the score went from 259.90 to -520.33 and stayed negative until the last episode 500.
episode: 468 score 248.95 average score 188.47 epsilon 0.01
episode: 469 score 269.47 average score 192.11 epsilon 0.01
episode: 470 score 225.31 average score 195.72 epsilon 0.01
episode: 471 score 46.36 average score 194.66 epsilon 0.01
episode: 472 score 250.61 average score 198.44 epsilon 0.01
episode: 473 score 237.25 average score 198.50 epsilon 0.01
episode: 474 score 237.02 average score 202.16 epsilon 0.01
episode: 475 score 249.85 average score 204.11 epsilon 0.01
episode: 476 score 259.90 average score 204.15 epsilon 0.01
episode: 477 score -520.33 average score 198.10 epsilon 0.01
episode: 478 score -567.86 average score 190.58 epsilon 0.01
episode: 479 score -670.45 average score 185.15 epsilon 0.01
episode: 480 score -413.31 average score 178.85 epsilon 0.01
episode: 481 score -995.13 average score 166.78 epsilon 0.01
episode: 482 score -450.73 average score 159.52 epsilon 0.01
episode: 483 score -583.07 average score 151.16 epsilon 0.01
episode: 484 score -586.59 average score 143.61 epsilon 0.01
episode: 485 score -436.80 average score 136.45 epsilon 0.01
episode: 486 score -665.69 average score 127.73 epsilon 0.01
episode: 487 score -602.49 average score 119.65 epsilon 0.01
episode: 488 score -1685.73 average score 99.95 epsilon 0.01
episode: 489 score -689.85 average score 91.23 epsilon 0.01
episode: 490 score -501.80 average score 83.63 epsilon 0.01
episode: 491 score -1016.14 average score 74.79 epsilon 0.01
episode: 492 score -475.65 average score 67.81 epsilon 0.01
episode: 493 score -525.98 average score 59.80 epsilon 0.01
episode: 494 score -393.08 average score 53.57 epsilon 0.01
episode: 495 score -557.32 average score 46.85 epsilon 0.01
episode: 496 score -431.39 average score 39.88 epsilon 0.01
episode: 497 score -944.19 average score 27.72 epsilon 0.01
episode: 498 score -571.92 average score 19.96 epsilon 0.01
episode: 499 score -498.09 average score 14.63 epsilon 0.01
What happened?
Is it overfitting?
Is it normal?
Does it mean that we have to run fewer episodes or run it again?
I'm confused about how to test different models and hyperparameters.
Here's the full code
import gym
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.optimizers import <PERSON><PHONE_NUMBER> average score 99.95 epsilon 0.01
episode: 489 score -689.85 average score 91.23 epsilon 0.01
episode: 490 score -501.80 average score 83.63 epsilon 0.01
episode: 491 score -1016.14 average score 74.79 epsilon 0.01
episode: 492 score -475.65 average score 67.81 epsilon 0.01
episode: 493 score -525.98 average score 59.80 epsilon 0.01
episode: 494 score -393.08 average score 53.57 epsilon 0.01
episode: 495 score -557.32 average score 46.85 epsilon 0.01
episode: 496 score -431.39 average score 39.88 epsilon 0.01
episode: 497 score -944.19 average score 27. 458