Answers.Body: <p>Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent <em>tanh</em> converges very quickly towards <span class="math-container">$-1$</span> or <span class="math-container">$1$</span>, so that is why you always find <span class="math-container">$-1$</span> and <span class="math-container">$1$</span> in the output.</p>
  <p>Let us execute the following code to get a better idea:</p>
  <pre><code>import tensorflow as tf
  tanh_x = tf.keras.activations.tanh(-8.0).numpy()
  print(tanh_x, type(tanh_x))
  </code></pre>
  <p>At least in my machine, the output will be exactly <span class="math-container">$1.0$</span> for <code>tanh_x</code> variable of type <code>numpy.float32</code>, which has a precision of up to <span class="math-container">$7$</span> decimal digits.</p>
  <p>The value of <code>tanh_x</code> is in fact <span class="math-container">$-0.99999977493$</span>, but Python only saves the first <span class="math-container">$7$</span> decimal digits and decides to round it up to <span class="math-container">$-1.0$</span>. Every output value from the neural network outside the range of <span class="math-container">$[-8, 8]$</span>  will be exactly <span class="math-container">$-1$</span> or <span class="math-container">$1$</span> after <em>tanh</em> activation and <em>float32</em> precision. As you can see, if the network randomly outputs values they will almost surely be outside <span class="math-container">$[-8, 8]$</span> as this is a very short range.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T20:14:39.927Z
Answers.Text: Thank you for the comment and example. Does it also mean the network is not well configured(not enough layers and neurons)?
Answers.UserDisplayName: goldfinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T21:34:41.260Z
Answers.Text: @Goldfinch Honestly, I think your network is too large with 120k parameters. You could begin by reducing dense layer sizes to 30 and 40 instead of 300 and 400.
Answers.UserDisplayName: devidduma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-21T13:17:39.160Z
Answers.LastActivityDate: 2022-02-21T13:54:42.647Z
Answers.LastEditDate: 2022-02-21T13:54:42.647Z
Answers.UserDisplayName: devidduma
Body: <p>I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,yaw). The action space is continuous too in the range (0, 1.57). I apply scaling at the input layer, and scale things back before applying the action received from network. Irrespective of learning for 100 or 1000 episodes, the actor model always gives either -1 or 1. Eg: [1,1,-1,-1] which gets scaled to [1.57,1.57,-1.57,-1.57] as an action vector. Could someone give me suggestion on what's happening with the network. The learning follows DDPG algorithm.</p>
  <pre><code>actor_lr = 0.0001
  critic_lr = 0.001
  def actor(state_size, action_size):
  
      inputs = Input(shape=(state_size,), name=&quot;state_space&quot;)
      layer_one = Dense(300, activation=&quot;relu&quot;)(inputs)
      layer_two = Dense(400, activation=&quot;relu&quot;)(layer_one)
  
      outputs = Dense(action_size, activation=&quot;tanh&quot;, name=&quot;action_space&quot;)(layer_two)
  
      model = Model(inputs = inputs, outputs = outputs)
      model.compile(loss=&quot;mse&quot;, optimizer=Adam(lr=actor_lr), metrics=[&quot;accuracy&quot;])
      return model
  
  def get_critic(state_size, action_size):
  
      state_input = Input(shape=(state_size), name=&quot;state_space&quot;)
      state_out = Dense(64, activation=&quot;relu&quot;)(state_input)
  
      action_input = Input(shape=(action_size), name=&quot;actions_sapce&quot;)
      action_out = Dense(18, activation=&quot;relu&quot;)(action_input)
      concat = Concatenate()([state_out, action_out])
      
      critic_layer_one = Dense(300, activation=&quot;relu&quot;, kernel_regularizer=regularizers.l2(0.01))(concat)
      critic_layer_two = Dense(400, activation=&quot;relu&quot;, kernel_regularizer=regularizers.l2(0.01))(critic_layer_one)
  
      outputs = Dense(1, activation=&quot;linear&quot;, kernel_regularizer=regularizers.l2(0.01))(critic_layer_two)
  
      model = Model([state_input, action_input], outputs, name = name)
      model.compile(loss=&quot;mse&quot;, optimizer=Adam(lr=critic_lr), metrics=[&quot;accuracy&quot;])
      return model
  </code></pre>
  
Tags: <reinforcement-learning><deep-rl><ddpg>
Title: Why does my actor-critic network always give either -1 or 1 at the output layer?
UserDisplayName: goldfinch
ThreadText: # Why does my actor-critic network always give either -1 or 1 at the output layer?
  
  (Asked by: username_0 on 2022-02-21)
  
  I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,yaw). The action space is continuous too in the range (0, 1.57). I apply scaling at the input layer, and scale things back before applying the action received from network. Irrespective of learning for 100 or 1000 episodes, the actor model always gives either -1 or 1. Eg: [1,1,-1,-1] which gets scaled to [1.57,1.57,-1.57,-1.57] as an action vector. Could someone give me suggestion on what's happening with the network. The learning follows DDPG algorithm.
  actor_lr = 0.0001
  critic_lr = 0.001
  def actor(state_size, action_size):
  
      inputs = Input(shape=(state_size,), name="state_space")
      layer_one = Dense(300, activation="relu")(inputs)
      layer_two = Dense(400, activation="relu")(layer_one)
  
      outputs = Dense(action_size, activation="tanh", name="action_space")(layer_two)
  
      model = Model(inputs = inputs, outputs = outputs)
      model.compile(loss="mse", optimizer=Adam(lr=actor_lr), metrics=["accuracy"])
      return model
  
  def get_critic(state_size, action_size):
  
      state_input = Input(shape=(state_size), name="state_space")
      state_out = Dense(64, activation="relu")(state_input)
  
      action_input = Input(shape=(action_size), name="actions_sapce")
      action_out = Dense(18, activation="relu")(action_input)
      concat = Concatenate()([state_out, action_out])
      
      critic_layer_one = Dense(300, activation="relu", kernel_regularizer=regularizers.l2(0.01))(concat)
      critic_layer_two = Dense(400, activation="relu", kernel_regularizer=regularizers.l2(0.01))(critic_layer_one)
  
      outputs = Dense(1, activation="linear", kernel_regularizer=regularizers.l2(0.01))(critic_layer_two)
  
      model = Model([state_input, action_input], outputs, name = name)
      model.compile(loss="mse", optimizer=Adam(lr=critic_lr), metrics=["accuracy"])
      return model
  
  **Tags:** reinforcement-learning, deep-rl, ddpg
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-02-21. Score: 1 | Accepted answer
  
  Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ and $1$ in the output.
  Let us execute the following code to get a better idea:
  import tensorflow as tf
  tanh_x = tf.keras.activations.tanh(-8.0).numpy()
  print(tanh_x, type(tanh_x))
  
  At least in my machine, the output will be exactly $1.0$ for tanh_x variable of type numpy.float32, which has a precision of up to $7$ decimal digits.
  The value of tanh_x is in fact $-0.99999977493$, but Python only saves the first $7$ decimal digits and decides to round it up to $-1.0$. Every output value from the neural network outside the range of $[-8, 8]$  will be exactly $-1$ or $1$ after tanh activation and float32 precision. As you can see, if the network randomly outputs values they will almost surely be outside $[-8, 8]$ as this is a very short range.
  
  > **Comments:**
  > **username_0**: Thank you for the comment and example. Does it also mean the network is not well configured(not enough layers and neurons)?  
  > **username_1**: @Goldfinch Honestly, I think your network is too large with 120k parameters. You could begin by reducing dense layer sizes to 30 and 40 instead of 300 and 400.  
  
  ---
  
  