Answers.Body: <p>After tinkering a bit more with my experiment, i got it to consistently manifest the intended behavior after around 200 episodes.</p>
  <p>Changes to the model itself were minimal: i replaced the loss function on the actor to <code>tf.losses.softmaxCrossEntropy</code>.</p>
  <p>Some changes to the training environment seemed to have a significant impact and improved training:</p>
  <ul>
  <li>Ending the episode after the reward reaches a minimum threshold - this prevents the models being polluted when the car was stock in a corner or flat against a wall.</li>
  <li>Making sure that the reward used in training was inline with the action that produced that reward - the model training in my case is asynchronous to the physics of the environment - i am sampling the simulation and providing inputs, but the simulation is not interrupted by model related processing.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-05T19:30:09.893Z
Answers.LastActivityDate: 2020-12-05T19:30:09.893Z
Answers.UserDisplayName: Sergiu Ionescu
Body: <p>I am trying to implement an Actor Critic method that controls an RC car. For this I have implemented a simulated environment and actor critic tensorflowjs models.</p>
  <p>My intention is to train a model to navigate an environment without colliding with various obstacles.</p>
  <p>For this I have the following:</p>
  <p>State(continuous):</p>
  <ul>
  <li>the sensors distance(left, middle, right): [0..1,0..1,0..1]</li>
  </ul>
  <p>Action(discrete):</p>
  <ul>
  <li>4 possible actions(move forward, move back, turn left, turn right)</li>
  </ul>
  <p>Reward(cumulative):</p>
  <ul>
  <li>moving forward is encouraged</li>
  <li>being close to an obstacle is penalized</li>
  <li>colliding with an obstacle is penalized</li>
  </ul>
  <p>The structure of the models:</p>
  <pre><code>buildActor() {
        const model = tf.sequential();
        model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits/2),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: this.actionSize,
          activation: 'softmax',
          kernelInitializer: 'glorotUniform',
        }));
  
        this.compile(model, this.actorLearningRate);
  
        return model;
      }
  </code></pre>
  <pre><code>buildCritic() {
        const model = tf.sequential();
  
        model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits/2),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: this.valueSize,
          activation: 'linear',
          kernelInitializer: 'glorotUniform',
        }));
  
        this.compile(model, this.criticLearningRate);
  
        return model;
      }
  </code></pre>
  <p>The models are compiled with an adam optimized and huber loss:</p>
  <pre><code>compile(model, learningRate) {
        model.compile({
          optimizer: tf.train.adam(learningRate),
          loss: tf.losses.huberLoss,
        });
      }
  </code></pre>
  <p>Training:</p>
  <pre><code>trainModel(state, action, reward, nextState) {
        let advantages = new Array(this.actionSize).fill(0);
  
        let normalizedState = normalizer.normalizeFeatures(state);
        let tfState = tf.tensor2d(normalizedState, [1, state.length]);
        let normalizedNextState = normalizer.normalizeFeatures(nextState);
        let tfNextState = tf.tensor2d(normalizedNextState, [1, nextState.length]);
  
        let predictedCurrentStateValue = this.critic.predict(tfState).dataSync();
        let predictedNextStateValue = this.critic.predict(tfNextState).dataSync();
  
        let target = reward + this.discountFactor * predictedNextStateValue;
        let advantage = target - predictedCurrentStateValue;
        advantages[action] = advantage;
        // console.log(normalizedState, normalizedNextState, action, target, advantages);
  
        this.actor.fit(tfState, tf.tensor([advantages]), {
          epochs: 1,
        }).then(info =&gt; {
            this.latestActorLoss = info.history.loss[0];
            this.actorLosses.push(this.latestActorLoss);
          }
        );
  
        this.critic.fit(tfState, tf.tensor([target]), {
          epochs: 1,
        }).then(info =&gt; {
            this.latestCriticLoss = info.history.loss[0];
            this.criticLosses.push(this.latestCriticLoss);
          }
        );
  
        this.advantages.push(advantage);
        pushToEvolutionChart(this.epoch, this.latestActorLoss, this.latestCriticLoss, advantage);
        this.epoch++;
      }
  </code></pre>
  <p>You ca give the simulation a spin on <a href="https://sergiuionescu.github.io/esp32-auto-car/sym/sym.html" rel="nofollow noreferrer">https://sergiuionescu.github.io/esp32-auto-car/sym/sym.html</a> .</p>
  <p>I found that some behaviors are being picked up - the model learns to prioritize moving forward after a few episodes, but then hits the wall and it reprioritizes spinning - but seems to completely 'forget' that moving forward was ever prioritized.</p>
  <p>I've been trying to follow <a href="https://keras.io/examples/rl/actor_critic_cartpole/" rel="nofollow noreferrer">https://keras.io/examples/rl/actor_critic_cartpole/</a> to a certain extent, but have not found an equivalent of the way back-propagation is handled there - GradientTape.</p>
  <p>Is it possible to perform training similar to the Keras example in Tensorflowjs?</p>
  <p>The theory i've went through on Actor Critic mentions that the Critic should estimate the reward yet to be obtain until the rest of the episode, but i am training the critic with:
  <code>reward + this.discountFactor * predictedNextStateValue</code> where <strong>reward</strong> is the cumulative reward until the current step.
  Should i keep track of a maximum total reward in previous episodes and subtract my reward from that instead?</p>
  <p>When i am training the actor i am generating a zero filled advantages tensor:</p>
  <pre><code>let advantages = new Array(this.actionSize).fill(0);
  let target = reward + this.discountFactor * predictedNextStateValue;
  let advantage = target - predictedCurrentStateValue;
  advantages[action] = advantage;
  </code></pre>
  <p>All other actions than the taken one will receive a 0 advantage. Could this discourage any previous actions the were proven beneficial?
  Should i average out the advantages per state and action?</p>
  <p>Thanks for having the patience to go trough all of this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T18:24:34.937Z
Comments.Text: I've been trying to follow https://greentec.github.io/reinforcement-learning-fourth-en/ which has an in depth explanation for the A2C method.
  
  It seems that the formula for calculating the advantage is in line with:
  `A(s,a)≃R+γV(s′)−V(s)`
  or `advantage = reward + this.discountFactor * predictedNextStateValue - predictedCurrentStateValue `
Comments.UserDisplayName: Sergiu Ionescu
Tags: <tensorflow><keras><actor-critic-methods>
Title: Advantage Actor Critic model implementation with Tensorflowjs
UserDisplayName: Sergiu Ionescu
ThreadText: # Advantage Actor Critic model implementation with Tensorflowjs
  
  (Asked by: username_0 on 2020-12-01)
  
  I am trying to implement an Actor Critic method that controls an RC car. For this I have implemented a simulated environment and actor critic tensorflowjs models.
  
  My intention is to train a model to navigate an environment without colliding with various obstacles.
  
  For this I have the following:
  
  State(continuous):
  
  * the sensors distance(left, middle, right): [0..1,0..1,0..1]
  
  Action(discrete):
  
  * 4 possible actions(move forward, move back, turn left, turn right)
  
  Reward(cumulative):
  
  * moving forward is encouraged
  * being close to an obstacle is penalized
  * colliding with an obstacle is penalized
  
  The structure of the models:
  
  ```
  buildActor() {
        const model = tf.sequential();
        model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits/2),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: this.actionSize,
          activation: 'softmax',
          kernelInitializer: 'glorotUniform',
        }));
  
        this.compile(model, this.actorLearningRate);
  
        return model;
      }
  
  ```
  
  ```
  buildCritic() {
        const model = tf.sequential();
  
        model.add(tf.layers.inputLayer({inputShape: [this.stateSize],}));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: parseInt(this.config.hiddenUnits/2),
          activation: 'relu',
          kernelInitializer: 'glorotUniform',
        }));
  
        model.add(tf.layers.dense({
          units: this.valueSize,
          activation: 'linear',
          kernelInitializer: 'glorotUniform',
        }));
  
        this.compile(model, this.criticLearningRate);
  
        return model;
      }
  
  ```
  
  The models are compiled with an adam optimized and huber loss:
  
  ```
  compile(model, learningRate) {
        model.compile({
          optimizer: tf.train.adam(learningRate),
          loss: tf.losses.huberLoss,
        });
      }
  
  ```
  
  Training:
  
  ```
  trainModel(state, action, reward, nextState) {
        let advantages = new Array(this.actionSize).fill(0);
  
        let normalizedState = normalizer.normalizeFeatures(state);
        let tfState = tf.tensor2d(normalizedState, [1, state.length]);
        let normalizedNextState = normalizer.normalizeFeatures(nextState);
        let tfNextState = tf.tensor2d(normalizedNextState, [1, nextState.length]);
  
        let predictedCurrentStateValue = this.critic.predict(tfState).dataSync();
        let predictedNextStateValue = this.critic.predict(tfNextState).dataSync();
  
        let target = reward + this.discountFactor * predictedNextStateValue;
        let advantage = target - predictedCurrentStateValue;
        advantages[action] = advantage;
        // console.log(normalizedState, normalizedNextState, action, target, advantages);
  
        this.actor.fit(tfState, tf.tensor([advantages]), {
          epochs: 1,
        }).then(info => {
            this.latestActorLoss = info.history.loss[0];
            this.actorLosses.push(this.latestActorLoss);
          }
        );
  
        this.critic.fit(tfState, tf.tensor([target]), {
          epochs: 1,
        }).then(info => {
            this.latestCriticLoss = info.history.loss[0];
            this.criticLosses.push(this.latestCriticLoss);
          }
        );
  
        this.advantages.push(advantage);
        pushToEvolutionChart(this.epoch, this.latestActorLoss, this.latestCriticLoss, advantage);
        this.epoch++;
      }
  
  ```
  
  You ca give the simulation a spin on <https://username_0.github.io/esp32-auto-car/sym/sym.html> .
  
  I found that some behaviors are being picked up - the model learns to prioritize moving forward after a few episodes, but then hits the wall and it reprioritizes spinning - but seems to completely 'forget' that moving forward was ever prioritized.
  
  I've been trying to follow <https://keras.io/examples/rl/actor_critic_cartpole/> to a certain extent, but have not found an equivalent of the way back-propagation is handled there - GradientTape.
  
  Is it possible to perform training similar to the Keras example in Tensorflowjs?
  
  The theory i've went through on Actor Critic mentions that the Critic should estimate the reward yet to be obtain until the rest of the episode, but i am training the critic with:
  `reward + this.discountFactor * predictedNextStateValue` where **reward** is the cumulative reward until the current step.
  Should i keep track of a maximum total reward in previous episodes and subtract my reward from that instead?
  
  When i am training the actor i am generating a zero filled advantages tensor:
  
  ```
  let advantages = new Array(this.actionSize).fill(0);
  let target = reward + this.discountFactor * predictedNextStateValue;
  let advantage = target - predictedCurrentStateValue;
  advantages[action] = advantage;
  
  ```
  
  All other actions than the taken one will receive a 0 advantage. Could this discourage any previous actions the were proven beneficial?
  Should i average out the advantages per state and action?
  
  Thanks for having the patience to go trough all of this.
  
  **Tags:** tensorflow, keras, actor-critic-methods
  
  > **Comments:**
  > **username_0**: I've been trying to follow https://greentec.github.io/reinforcement-learning-fourth-en/ which has an in depth explanation for the A2C method.  It seems that the formula for calculating the advantage is in line with: `A(s,a)≃R+γV(s′)−V(s)` or `advantage = reward + this.discountFactor * predictedNextStateValue - predictedCurrentStateValue `  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-05. Score: 1
  
  After tinkering a bit more with my experiment, i got it to consistently manifest the intended behavior after around 200 episodes.
  
  Changes to the model itself were minimal: i replaced the loss function on the actor to `tf.losses.softmaxCrossEntropy`.
  
  Some changes to the training environment seemed to have a significant impact and improved training:
  
  * Ending the episode after the reward reaches a minimum threshold - this prevents the models being polluted when the car was stock in a corner or flat against a wall.
  * Making sure that the reward used in training was inline with the action that produced that reward - the model training in my case is asynchronous to the physics of the environment - i am sampling the simulation and providing inputs, but the simulation is not interrupted by model related processing.
  
  
  ---
  
  