Body: <p>I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the exact same DDPG to solve Pendulum-v0 and it works, but my environment is a more complex with a continuous space/action space.</p>
  <p>How do you deal with local optima problem in reinforcement learning? is it just an exploitation issue?</p>
  <p><strong>More details:</strong></p>
  <p>My state space is not pixels, it is numerical, in fact it's a metro line simulator and the state space is the velocity, the position of each train on the line and the number of passengers at each station. I need to control the different trains so I am not trying to control only one train but all the operational trains and each one can have different actions such as speed or not, stay longer on the next station or not etc.</p>
  <p>1/ I am using the same ANN architecture for the actor and critic: 3 FC layers with (512, 256, 256) hidden units.</p>
  <p>2/ Adam optimizer for both the actor and critic with a small lr=1e-7 and clipnorm=1.</p>
  <p>3/ nb_steps_warmup_critic=1000, nb_steps_warmup_actor=1000</p>
  <p>4/ SequentialMemory(limit=1000000, window_length=1)</p>
  <p>5/ The environement is a simulator of a metro line with a continuous state and action space</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-18T15:10:11.517Z
Comments.Text: Hi @BAKYAC. Given that there's no logical bug in the algorithm/program, it could come down to fine-tuning the hyperparameters (learning rate, ann architecture, etc.). As I understand, finding a perfect combination of those can be quite challenging. There are some vague general guidelines, but still, the hyperparameters are usually specific for each particular environment. If you elaborate in your question on what kind of architecture you have, and what kind of problem you're trying to solve, you might get some input from other users (though this SE is somewhat in dormant state).
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-19T13:44:20.893Z
Comments.Text: Thank you for your suggestions! I edited my question and yes I do think it's a hyperparameters fine-tuning problem but as the training takes too long I am not able to test out different parameters !
Comments.UserDisplayName: BAKYAC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-19T20:20:12.727Z
Comments.Text: You could try using Google Colab – they let you use their GPU and TPU for free, though there's a limit. That really helped me to speed up model training and tuning
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T10:13:22.497Z
Comments.Text: I have an NVIDIA Quadro T1000 GPU so I don't think I need google colab as they provide a tesla k80, plus the training crash after some time and mine can take days to finish..
Comments.UserDisplayName: BAKYAC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T20:01:36.630Z
Comments.Text: Got it. Is your environment visual or just numeric? Also, what are the nb_steps_warmup_critic and nb_steps_warmup_actor parameters?
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T20:03:55.880Z
Comments.Text: I see you're clipping the gradients, you could also look into adding regularization to your layers (that supposed to help the model to generalize and not to over-fit)
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T20:49:35.867Z
Comments.Text: My environement is visual with a continuous state space, I think that DDPG is so simple and can't solve it I will try TRPO or can you suggest any other algorithm?
Comments.UserDisplayName: BAKYAC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T21:01:05.803Z
Comments.Text: If you're working with pixels, you might need to add some convolutional layers, as they work better for visual feature recognition. Can't suggest any continuous algorithms unfortunately -- haven't worked with them before. This guide says that SAC and TD3 are current state-of-art algorithms: https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T21:18:34.957Z
Comments.Text: Oh no I am not working with pixels. I edited my question.. thanks anyway !
Comments.UserDisplayName: BAKYAC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-20T21:29:01.077Z
Comments.Text: and good luck with your project :)
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-21T08:14:29.823Z
Comments.Text: Thanks! will update this thread if I find something interesting..
Comments.UserDisplayName: BAKYAC
Tags: <reinforcement-learning><keras><open-ai><gym><ddpg>
Title: How to deal with KerasRL DDPG algorithm getting stuck in a local optima?
UserDisplayName: BAKYAC
ThreadText: # How to deal with KerasRL DDPG algorithm getting stuck in a local optima?
  
  (Asked by: username_0 on 2020-12-17)
  
  I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the exact same DDPG to solve Pendulum-v0 and it works, but my environment is a more complex with a continuous space/action space.
  
  How do you deal with local optima problem in reinforcement learning? is it just an exploitation issue?
  
  **More details:**
  
  My state space is not pixels, it is numerical, in fact it's a metro line simulator and the state space is the velocity, the position of each train on the line and the number of passengers at each station. I need to control the different trains so I am not trying to control only one train but all the operational trains and each one can have different actions such as speed or not, stay longer on the next station or not etc.
  
  1/ I am using the same ANN architecture for the actor and critic: 3 FC layers with (512, 256, 256) hidden units.
  
  2/ Adam optimizer for both the actor and critic with a small lr=1e-7 and clipnorm=1.
  
  3/ nb\_steps\_warmup\_critic=1000, nb\_steps\_warmup\_actor=1000
  
  4/ SequentialMemory(limit=1000000, window\_length=1)
  
  5/ The environement is a simulator of a metro line with a continuous state and action space
  
  **Tags:** reinforcement-learning, keras, open-ai, gym, ddpg
  
  > **Comments:**
  > **username_1**: Hi @username_0. Given that there's no logical bug in the algorithm/program, it could come down to fine-tuning the hyperparameters (learning rate, ann architecture, etc.). As I understand, finding a perfect combination of those can be quite challenging. There are some vague general guidelines, but still, the hyperparameters are usually specific for each particular environment. If you elaborate in your question on what kind of architecture you have, and what kind of problem you're trying to solve, you might get some input from other users (though this SE is somewhat in dormant state).  
  > **username_0**: Thank you for your suggestions! I edited my question and yes I do think it's a hyperparameters fine-tuning problem but as the training takes too long I am not able to test out different parameters !  
  > **username_1**: You could try using Google Colab – they let you use their GPU and TPU for free, though there's a limit. That really helped me to speed up model training and tuning  
  > **username_0**: I have an NVIDIA Quadro T1000 GPU so I don't think I need google colab as they provide a tesla k80, plus the training crash after some time and mine can take days to finish..  
  > **username_1**: Got it. Is your environment visual or just numeric? Also, what are the nb_steps_warmup_critic and nb_steps_warmup_actor parameters?  
  > **username_1**: I see you're clipping the gradients, you could also look into adding regularization to your layers (that supposed to help the model to generalize and not to over-fit)  
  > **username_0**: My environement is visual with a continuous state space, I think that DDPG is so simple and can't solve it I will try TRPO or can you suggest any other algorithm?  
  > **username_1**: If you're working with pixels, you might need to add some convolutional layers, as they work better for visual feature recognition. Can't suggest any continuous algorithms unfortunately -- haven't worked with them before. This guide says that SAC and TD3 are current state-of-art algorithms: https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html  
  > **username_0**: Oh no I am not working with pixels. I edited my question.. thanks anyway !  
  > **username_1**: and good luck with your project :)  
  > **username_0**: Thanks! will update this thread if I find something interesting..  
  
  ---
  
  ## 0 answers
  
  