Answers.Body: <p>The primary issue I was having was that I wasn't normalizing the input data before sending it through the system. I can confidently say that it is working now.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T00:09:14.937Z
Answers.LastActivityDate: 2023-11-20T00:09:14.937Z
Answers.UserDisplayName: CloudZero
Body: <p>I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to make a basic blueprint for myself and the community to use. I've put in a lot of work and I feel like it should basically be complete, but I can't find out why the agent keeps (very quickly) converging on a single action selection (even though it's being penalized for doing so, even testing at 300 iterations). I checked the loss variables and they seem to be working as expected. Although I'm great with JavaScript and have a solid understanding of how everything works, I'm not a professional with Python or TensorFlow.js (yet).</p>
  <p>I need some people with experience in the following areas to review my code: Reinforcement Learning, TD3 (or DDPG), Tensorflow.JS.</p>
  <p>Everything is on GitHub at <a href="https://github.com/CloudZero2049/TD3-TensorFlowJS" rel="nofollow noreferrer">https://github.com/CloudZero2049/TD3-TensorFlowJS</a></p>
  <p>All the info about the project is in the README. TD3script.js is the only one to worry about, there's also one without comments.</p>
  <p>I have been reviewing the code and math extensively everything seems to be working properly. But when testing the agent quickly settles on a single bad action even though it's being penalized. I tried 300 iterations, 200 warmup, 100 batch_size to see if it was a time issue without a change. I feel confident in the structure and math but something is clearly wrong. After going over the code so many times and making improvements I've finally hit the wall, I don't see how I can proceed.</p>
  <p>I expected at some point for the agent to realize it was doing something wrong and try something new. I would have gotten much better results if the actions were completely random.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T10:44:52.973Z
Comments.Text: One thing to check is whether you are normalising the NN inputs, it's a common omission for beginners. Unlikely anyone is going to review your project code for you, I suggest you reach out to the community you are hoping to serve with the js conversion.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-11T21:04:39.263Z
Comments.Text: I'm trying this now, thank you. I learned this early on and completely forgot. Now I'm not sure if it's better to normalize states and rewards before or after they are sent to the memory buffer. I'm trying "before" first because it's much easier to do.
Comments.UserDisplayName: CloudZero
Tags: <reinforcement-learning><machine-learning><actor-critic-methods><javascript><td3>
Title: Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?
UserDisplayName: CloudZero
ThreadText: # Why is my agent stuck on the same action in my Twin Delayed Deep Deterministic Policy Gradient (TD3) program?
  
  (Asked by: username_0 on 2023-11-11)
  
  I've been tirelessly converting a reinforcement learning program from Python to JavaScript using TensorFlow.js that is running Twin Delayed Deep Deterministic Policy Gradient (TD3). I'm just trying to make a basic blueprint for myself and the community to use. I've put in a lot of work and I feel like it should basically be complete, but I can't find out why the agent keeps (very quickly) converging on a single action selection (even though it's being penalized for doing so, even testing at 300 iterations). I checked the loss variables and they seem to be working as expected. Although I'm great with JavaScript and have a solid understanding of how everything works, I'm not a professional with Python or TensorFlow.js (yet).
  
  I need some people with experience in the following areas to review my code: Reinforcement Learning, TD3 (or DDPG), Tensorflow.JS.
  
  Everything is on GitHub at <https://github.com/CloudZero2049/TD3-TensorFlowJS>
  
  All the info about the project is in the README. TD3script.js is the only one to worry about, there's also one without comments.
  
  I have been reviewing the code and math extensively everything seems to be working properly. But when testing the agent quickly settles on a single bad action even though it's being penalized. I tried 300 iterations, 200 warmup, 100 batch\_size to see if it was a time issue without a change. I feel confident in the structure and math but something is clearly wrong. After going over the code so many times and making improvements I've finally hit the wall, I don't see how I can proceed.
  
  I expected at some point for the agent to realize it was doing something wrong and try something new. I would have gotten much better results if the actions were completely random.
  
  **Tags:** reinforcement-learning, machine-learning, actor-critic-methods, javascript, td3
  
  > **Comments:**
  > **username_1**: One thing to check is whether you are normalising the NN inputs, it's a common omission for beginners. Unlikely anyone is going to review your project code for you, I suggest you reach out to the community you are hoping to serve with the js conversion.  
  > **username_0**: I'm trying this now, thank you. I learned this early on and completely forgot. Now I'm not sure if it's better to normalize states and rewards before or after they are sent to the memory buffer. I'm trying "before" first because it's much easier to do.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-11-20. Score: 0
  
  The primary issue I was having was that I wasn't normalizing the input data before sending it through the system. I can confidently say that it is working now.
  
  
  ---
  
  