Answers.Body: <p>Just use one class inheriting from <code>nn.Module</code> called e.g. <code>ActorCriticModel</code>. </p>
  
  <p>Then, have two members called <code>self.actor</code> and <code>self.critic</code> and define them to have the desired architecture.Then, in the <code>forward()</code> method return two values, one for the actor output (which is a vector) and one for the critic value (which is a scalar). </p>
  
  <p>This way you can use only one optimizer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-28T09:26:18.530Z
Answers.LastActivityDate: 2019-10-28T09:26:18.530Z
Answers.UserDisplayName: Gabizon
Answers.Body: <p>A common practice involves using a shared encoder, which is updated based solely on critic loss, as implemented in <a href="https://github.com/facebookresearch/drqv2/blob/c0c650b76c6e5d22a7eb5f2edffd1440fe94f8ef/drqv2.py#L198" rel="nofollow noreferrer">DrQv2</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-13T04:20:06.647Z
Answers.LastActivityDate: 2023-11-13T04:20:06.647Z
Answers.UserDisplayName: XiaoBanni
Body: <p>I'm building an actor-critic reinforcement learning algorithm to solve environments. I want to use a single encoder to find representation of my environment.</p>
  <p>When I share the encoder with the actor and the critic, my network isn't learning anything:</p>
  <pre><code>class Encoder(nn.Module):
    def __init__(self, state_dim):
      super(Encoder, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 512)
  
    def forward(self, state):
      a = F.relu(self.l1(state))
      return a
  
  class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
      super(Actor, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 128)
      self.l3 = nn.Linear(128, action_dim)
  
      self.max_action = max_action
  
    def forward(self, state):
      a = F.relu(self.l1(state))
      # a = F.relu(self.l2(a))
      a = torch.tanh(self.l3(a)) * self.max_action
      return a
  
  class Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
      super(Critic, self).__init__()
  
      self.l1 = nn.Linear(state_dim + action_dim, 128)
      self.l3 = nn.Linear(128, 1)
  
    def forward(self, state, action):
      state_action = torch.cat([state, action], 1)
  
      q = F.relu(self.l1(state_action))
      # q = F.relu(self.l2(q))
      q = self.l3(q)
      return q
  </code></pre>
  <p>However, when I use different encoder for the actor and different for the critic, it learn properly.</p>
  <pre><code>class Actor(nn.Module):
  def __init__(self, state_dim, action_dim, max_action):
      super(Actor, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 400)
      self.l2 = nn.Linear(400, 300)
      self.l3 = nn.Linear(300, action_dim)
  
      self.max_action = max_action
  
  def forward(self, state):
      a = F.relu(self.l1(state))
      a = F.relu(self.l2(a))
      a = torch.tanh(self.l3(a)) * self.max_action
      return a
  
  class Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
      super(Critic, self).__init__()
  
      self.l1 = nn.Linear(state_dim + action_dim, 400)
      self.l2 = nn.Linear(400, 300)
      self.l3 = nn.Linear(300, 1)
  
    def forward(self, state, action):
      state_action = torch.cat([state, action], 1)
  
      q = F.relu(self.l1(state_action))
      q = F.relu(self.l2(q))
      q = self.l3(q)
      return q
  </code></pre>
  <p>I'm pretty sure its because of the optimizer. In the shared encoder code, I define it as follows:</p>
  <pre><code>self.actor_optimizer = optim.Adam(list(self.actor.parameters())+
                                        list(self.encoder.parameters()))
  self.critic_optimizer = optim.Adam(list(self.critic.parameters()))
                                           +list(self.encoder.parameters()))
  </code></pre>
  <p>In the separate encoder, its just:</p>
  <pre><code>self.actor_optimizer = optim.Adam((self.actor.parameters()))
  self.critic_optimizer = optim.Adam((self.critic.parameters()))
  </code></pre>
  <p>two optimizers must be because of the actor critic algorithm, in which the loss of the actor is the value.</p>
  <p>How can I combine two optimizers to optimize correctly the encoder?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-28T21:40:52.317Z
Comments.Text: I’m not sure I understand yet; are you using the encoder to transform the state representation to a new representation, and then feeding this new representation to the actor and critic networks? Also, where are you sharing the encoder in the code for the actor / critic? I didn’t see where exactly.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T15:43:46.507Z
Comments.Text: @Hanzy yes I use encoder to create shared representation between actor and critic.
  I just call enc=encoder(x)->actor(enc)/critic(enc)
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T16:20:13.740Z
Comments.Text: I don’t understand why you want to train the encoder this way? Why not just separately train an autoencoder and then use the trained autoencoder to produce a representation that you send to the actor and critic? Also, why not just send it a raw representation? Just curious. Maybe I misunderstand the motivation here.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T17:30:25.783Z
Comments.Text: Shouldn’t the encoder be evaluated (and updated) based on how accurately it encodes the information it’s given rather than the $Q$ values of different state action pairs? By updating a Q value, why update the representation?
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T17:36:49.097Z
Comments.Text: How do I train seperatly autoencoder?
  What is the output for training?
  And I send raw representation to the encoder, which I want to be updated both by actor and critic to get better results in representing the data.
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T17:46:31.123Z
Comments.Text: The output of the encoder will be the compressed representation of the raw input you feed to it. This blog post https://medium.com/@s.ganjoo96/autoencoders-with-pytorch-a89ed28f94a9 may help. But, I don’t think you need an encoder. Can I ask why not feed it a raw representation of your data? Why not just feed whatever you were putting into your encoder directly to the actor & critic?
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-01T18:49:08.600Z
Comments.Text: I can feed it but I can miss data. The shared laye construct some other unknown representation which is useful for all the prediction tasks.
  In the link you gave me there is an output-its like image to image. Here I dont have output to the representation
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T09:39:14.043Z
Comments.Text: @Hanzy And lets assume I have good reasons. Do you have an idea how to do it on pytorch?
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:26:28.073Z
Comments.Text: I have seen implementations that create an embedding of state space via CNN or other architecture, but I believe you need to decide WHAT you want that embedding to capture. For example, people have used it to determine an internal reward function, which seems similar to a value network. But they are always trained separately. However...
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:29:29.723Z
Comments.Text: If you simply want to make a new representation with an auto encoder: feed the representation (raw) to an encoder network. Use the output of that to feed into a decoder network. The output shape of decoder network should have the same shape as the input to the encoder network. Use MSE as a loss to minimize between the input and output. The goal is to recreate as close to perfectly as possible the original input to the encoder (thus minimize MSE between encoder input and decoder output). When this has been trained, you can use the encoder output as input to your AC network.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:31:00.847Z
Comments.Text: But I don’t know if this will be beneficial FYI. Let me know if you try and it does help.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:34:05.590Z
Comments.Text: @Hanzy this process just be the identity network. What will I do with the decoder?
  The encoder will just output the input, and same the decoder. How do I combine the learning with the AC networks?
Comments.UserDisplayName: BestR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:41:46.513Z
Comments.Text: the encoder should have a different dimensionally output than input. But yes, you are learning a new representation and then reconstruction of your state space. It won’t be that helpful because no new information has been embedded into the new representation. It’s not clear what you are trying to embed into the new representation.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-03T12:47:26.493Z
Comments.Text: I want the network to learn the new representation based on the losses from the critic and the actor..
Comments.UserDisplayName: BestR
Tags: <neural-networks><reinforcement-learning><optimization><actor-critic-methods><pytorch>
Title: How to properly optimize shared network between actor and critic?
UserDisplayName: BestR
ThreadText: # How to properly optimize shared network between actor and critic?
  
  (Asked by: username_0 on 2019-04-25)
  
  I'm building an actor-critic reinforcement learning algorithm to solve environments. I want to use a single encoder to find representation of my environment.
  When I share the encoder with the actor and the critic, my network isn't learning anything:
  class Encoder(nn.Module):
    def __init__(self, state_dim):
      super(Encoder, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 512)
  
    def forward(self, state):
      a = F.relu(self.l1(state))
      return a
  
  class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
      super(Actor, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 128)
      self.l3 = nn.Linear(128, action_dim)
  
      self.max_action = max_action
  
    def forward(self, state):
      a = F.relu(self.l1(state))
      # a = F.relu(self.l2(a))
      a = torch.tanh(self.l3(a)) * self.max_action
      return a
  
  class Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
      super(Critic, self).__init__()
  
      self.l1 = nn.Linear(state_dim + action_dim, 128)
      self.l3 = nn.Linear(128, 1)
  
    def forward(self, state, action):
      state_action = torch.cat([state, action], 1)
  
      q = F.relu(self.l1(state_action))
      # q = F.relu(self.l2(q))
      q = self.l3(q)
      return q
  
  However, when I use different encoder for the actor and different for the critic, it learn properly.
  class Actor(nn.Module):
  def __init__(self, state_dim, action_dim, max_action):
      super(Actor, self).__init__()
  
      self.l1 = nn.Linear(state_dim, 400)
      self.l2 = nn.Linear(400, 300)
      self.l3 = nn.Linear(300, action_dim)
  
      self.max_action = max_action
  
  def forward(self, state):
      a = F.relu(self.l1(state))
      a = F.relu(self.l2(a))
      a = torch.tanh(self.l3(a)) * self.max_action
      return a
  
  class Critic(nn.Module):
    def __init__(self, state_dim, action_dim):
      super(Critic, self).__init__()
  
      self.l1 = nn.Linear(state_dim + action_dim, 400)
      self.l2 = nn.Linear(400, 300)
      self.l3 = nn.Linear(300, 1)
  
    def forward(self, state, action):
      state_action = torch.cat([state, action], 1)
  
      q = F.relu(self.l1(state_action))
      q = F.relu(self.l2(q))
      q = self.l3(q)
      return q
  
  I'm pretty sure its because of the optimizer. In the shared encoder code, I define it as follows:
  self.actor_optimizer = optim.Adam(list(self.actor.parameters())+
                                        list(self.encoder.parameters()))
  self.critic_optimizer = optim.Adam(list(self.critic.parameters()))
                                           +list(self.encoder.parameters()))
  
  In the separate encoder, its just:
  self.actor_optimizer = optim.Adam((self.actor.parameters()))
  self.critic_optimizer = optim.Adam((self.critic.parameters()))
  
  two optimizers must be because of the actor critic algorithm, in which the loss of the actor is the value.
  How can I combine two optimizers to optimize correctly the encoder?
  
  **Tags:** neural-networks, reinforcement-learning, optimization, actor-critic-methods, pytorch
  > **Comments:**
  > **username_1**: I’m not sure I understand yet; are you using the encoder to transform the state representation to a new representation, and then feeding this new representation to the actor and critic networks? Also, where are you sharing the encoder in the code for the actor / critic? I didn’t see where exactly.  
  > **username_0**: @Hanzy yes I use encoder to create shared representation between actor and critic. I just call enc=encoder(x)->actor(enc)/critic(enc)  
  > **username_1**: I don’t understand why you want to train the encoder this way? Why not just separately train an autoencoder and then use the trained autoencoder to produce a representation that you send to the actor and critic? Also, why not just send it a raw representation? Just curious. Maybe I misunderstand the motivation here.  
  > **username_1**: Shouldn’t the encoder be evaluated (and updated) based on how accurately it encodes the information it’s given rather than the $Q$ values of different state action pairs? By updating a Q value, why update the representation?  
  > **username_0**: How do I train seperatly autoencoder? What is the output for training? And I send raw representation to the encoder, which I want to be updated both by actor and critic to get better results in representing the data.  
  > **username_1**: The output of the encoder will be the compressed representation of the raw input you feed to it. This blog post https://medium.com/@s.ganjoo96/autoencoders-with-pytorch-a89ed28f94a9 may help. But, I don’t think you need an encoder. Can I ask why not feed it a raw representation of your data? Why not just feed whatever you were putting into your encoder directly to the actor & critic?  
  > **username_0**: I can feed it but I can miss data. The shared laye construct some other unknown representation which is useful for all the prediction tasks. In the link you gave me there is an output-its like image to image. Here I dont have output to the representation  
  > **username_0**: @Hanzy And lets assume I have good reasons. Do you have an idea how to do it on pytorch?  
  > **username_1**: I have seen implementations that create an embedding of state space via CNN or other architecture, but I believe you need to decide WHAT you want that embedding to capture. For example, people have used it to determine an internal reward function, which seems similar to a value network. But they are always trained separately. However...  
  > **username_1**: If you simply want to make a new representation with an auto encoder: feed the representation (raw) to an encoder network. Use the output of that to feed into a decoder network. The output shape of decoder network should have the same shape as the input to the encoder network. Use MSE as a loss to minimize between the input and output. The goal is to recreate as close to perfectly as possible the original input to the encoder (thus minimize MSE between encoder input and decoder output). When this has been trained, you can use the encoder output as input to your AC network.  
  > **username_1**: But I don’t know if this will be beneficial FYI. Let me know if you try and it does help.  
  > **username_0**: @Hanzy this process just be the identity network. What will I do with the decoder? The encoder will just output the input, and same the decoder. How do I combine the learning with the AC networks?  
  > **username_1**: the encoder should have a different dimensionally output than input. But yes, you are learning a new representation and then reconstruction of your state space. It won’t be that helpful because no new information has been embedded into the new representation. It’s not clear what you are trying to embed into the new representation.  
  > **username_0**: I want the network to learn the new representation based on the losses from the critic and the actor..  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-28. Score: 0
  
  Just use one class inheriting from `nn.Module` called e.g. `ActorCriticModel`.
  
  Then, have two members called `self.actor` and `self.critic` and define them to have the desired architecture.Then, in the `forward()` method return two values, one for the actor output (which is a vector) and one for the critic value (which is a scalar).
  
  This way you can use only one optimizer.
  
  
  ---
  
  ### Answer by username_3 on 2023-11-13. Score: 0
  
  A common practice involves using a shared encoder, which is updated based solely on critic loss, as implemented in [DrQv2](https://github.com/facebookresearch/drqv2/blob/c0c650b76c6e5d22a7eb5f2edffd1440fe94f8ef/drqv2.py#L198).
  
  
  ---
  
  