Body: <p>As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit.</p>
  <p>My action vector is perfectly described as a vector, where all values are between 0 and 1, and which should sum up to 1. This is perfect for a Softmax function. But these values are not probabilities of discrete actions. Each value in action vector should be a percentage from the whole portfolio to be invested in a certain stock.</p>
  <p>But I cannot figure out, if it would be mathematically fine to use Softmax as an activation layer in TD3 or SAC?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-09T23:12:44.053Z
Comments.Text: You should check out discrete SAC. Original SAC was designed for continuous action spaces (the code was, at least — I believe theoretically the paper works with both but was motivated by continuous action spaces) but I believe they developed a discrete version. I don’t have the reference to hand but a Google search should turn up results.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T07:53:22.007Z
Comments.Text: @DavidIreland, my action space is not discrete actually. I did not imply probabilities by these values. Each value in action vector is actually a percentage from the whole portfolio to be invested in a certain stock.
Comments.UserDisplayName: Bi0max
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T12:31:25.143Z
Comments.Text: Ah, okay, I apologise. In which case then you can just use SAC with a soft max activation. I believe tanh is usually used because they have symmetric intervals for the action space, it is not something it relies on theoretically.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T13:08:35.863Z
Comments.Text: @DavidIreland, thank you for the answer. I also assumed so, but I was not sure whether it's mathematically fine or not.
Comments.UserDisplayName: Bi0max
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T16:15:29.893Z
Comments.Text: Yes, it should be fine :)
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><activation-functions><soft-actor-critic><td3>
Title: Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?
UserDisplayName: Bi0max
ThreadText: # Is it possible to use Softmax as an activation function for actor (policy) network in TD3 or SAC Reinforcement learning algorithms?
  
  (Asked by: username_0 on 2021-08-09)
  
  As I understand from literature, normally, the last activation in an actor (policy) network in TD3 and SAC algorithms is a Tanh function, which is scaled by a certain limit.
  
  My action vector is perfectly described as a vector, where all values are between 0 and 1, and which should sum up to 1. This is perfect for a Softmax function. But these values are not probabilities of discrete actions. Each value in action vector should be a percentage from the whole portfolio to be invested in a certain stock.
  
  But I cannot figure out, if it would be mathematically fine to use Softmax as an activation layer in TD3 or SAC?
  
  **Tags:** reinforcement-learning, deep-rl, activation-functions, soft-actor-critic, td3
  
  > **Comments:**
  > **username_1**: You should check out discrete SAC. Original SAC was designed for continuous action spaces (the code was, at least — I believe theoretically the paper works with both but was motivated by continuous action spaces) but I believe they developed a discrete version. I don’t have the reference to hand but a Google search should turn up results.  
  > **username_0**: @DavidIreland, my action space is not discrete actually. I did not imply probabilities by these values. Each value in action vector is actually a percentage from the whole portfolio to be invested in a certain stock.  
  > **username_1**: Ah, okay, I apologise. In which case then you can just use SAC with a soft max activation. I believe tanh is usually used because they have symmetric intervals for the action space, it is not something it relies on theoretically.  
  > **username_0**: @DavidIreland, thank you for the answer. I also assumed so, but I was not sure whether it's mathematically fine or not.  
  > **username_1**: Yes, it should be fine :)  
  
  ---
  
  ## 0 answers
  
  