Body: <p>I want to build a Deep Reinforcement Learning Model for Asset allocation.</p>
  <h1>Background:</h1>
  <p>I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether to sell or buy index? which index? and how much?) by observing the market informations.</p>
  <h1>Question 1:</h1>
  <p>I have two idea for the output of my policy. One is to produce a vector <span class="math-container">$w$</span> of length 8, Each element <span class="math-container">$w_i$</span> represent the target ratio of the stocks we want to hold (7 stock indexes and 1 cash), so I need to set <span class="math-container">$w_i&gt;0, $</span> and <span class="math-container">$\sum_{i}^{8}w_i=1.$</span> How to implement? I just let the Activation function in the last layer of neural network to be sigmoid and divide the sum in environment. Is this available? And it's not easy to deal with transaction process if buy and sell fee exist. And the training process slowly when I use the policy gradient.</p>
  <p>The two is also produce a vector <span class="math-container">$w$</span> of length 8, For each element <span class="math-container">$w_i$</span> represent sell percent for stock i when <span class="math-container">$w_i$</span> is negative and buy percent of cash when <span class="math-container">$w_i$</span> is positive. It can solve the problem I meet in idea one. But I will meet a new question is cash is finite. I need to decide order of buy, in other words, which stock to buy first and buy which one later.</p>
  <h1>Question 2:</h1>
  <p>Many papers tell me to produce Distributed parameters by policy then create the action by distribution (like: normal distribution). It makes that more difficult to control the action  satisfy the condition above.</p>
  <p>And the result whether be unstable if the action is produce by sample.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-25T08:44:09.337Z
Comments.Text: Hi and welcome to AI stack exchange. Your two questions are linked, but it is a bit much to answer them in one place. They could be separated, and an answer given for the first one, if you explain which deep RL method you want to implement. That would be required for both Q1 and Q2 in any case, since how you represent your policy, and how you allow for exploration will both depend on this.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-26T02:38:20.807Z
Comments.Text: Thanks @NeilSlater for your reply, may be I have not express my question clearly. I have already written an environment [env](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/environment/DNN_MarketEnv.py) by idea one. The Observed data is just the daily data of market like [data](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/data/create_feature.csv). And here is my policy gradient [pg](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/PolicyGradient.py).
Comments.UserDisplayName: C.M. Cai
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-26T02:38:51.860Z
Comments.Text: I want to train a policy by reinforcement learning to improve or exceed the traditional assets portfolio policy. Which use the rolling quantile of PE(the Price Earnings Ratio) of Stocks indexs as the target ratio monthly. The PE is one column in my data.
  My initial idea was naive. I want to use multiple dense layers to get the actions, and then get the evaluation function(means the probability of this action) by multiple dense layers after actions. But the result is totally decided by the initialization and the training process is also slowly. I think lack of explore strategy is the reason.
Comments.UserDisplayName: C.M. Cai
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-26T09:53:14.090Z
Comments.Text: Could you confirm which policy gradient (PG) method you are attempting to use? PG is not a method by itself, it is the base theory that is used to create PG methods. Examples of PG methods are REINFORCE, Actor-Critic, PPO, DDPG.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><policy-gradients>
Title: How to set the multiple continuous actions with constraints
UserDisplayName: C.M. Cai
ThreadText: # How to set the multiple continuous actions with constraints
  
  (Asked by: username_0 on 2019-11-25)
  
  I want to build a Deep Reinforcement Learning Model for Asset allocation.
  
  # Background:
  
  I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether to sell or buy index? which index? and how much?) by observing the market informations.
  
  # Question 1:
  
  I have two idea for the output of my policy. One is to produce a vector $w$ of length 8, Each element $w\_i$ represent the target ratio of the stocks we want to hold (7 stock indexes and 1 cash), so I need to set $w\_i>0, $ and $\sum\_{i}^{8}w\_i=1.$ How to implement? I just let the Activation function in the last layer of neural network to be sigmoid and divide the sum in environment. Is this available? And it's not easy to deal with transaction process if buy and sell fee exist. And the training process slowly when I use the policy gradient.
  
  The two is also produce a vector $w$ of length 8, For each element $w\_i$ represent sell percent for stock i when $w\_i$ is negative and buy percent of cash when $w\_i$ is positive. It can solve the problem I meet in idea one. But I will meet a new question is cash is finite. I need to decide order of buy, in other words, which stock to buy first and buy which one later.
  
  # Question 2:
  
  Many papers tell me to produce Distributed parameters by policy then create the action by distribution (like: normal distribution). It makes that more difficult to control the action satisfy the condition above.
  
  And the result whether be unstable if the action is produce by sample.
  
  **Tags:** reinforcement-learning, policy-gradients
  > **Comments:**
  > **username_1**: Hi and welcome to AI stack exchange. Your two questions are linked, but it is a bit much to answer them in one place. They could be separated, and an answer given for the first one, if you explain which deep RL method you want to implement. That would be required for both Q1 and Q2 in any case, since how you represent your policy, and how you allow for exploration will both depend on this.  
  > **username_0**: Thanks @NeilSlater for your reply, may be I have not express my question clearly. I have already written an environment [env](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/environment/DNN_MarketEnv.py) by idea one. The Observed data is just the daily data of market like [data](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/data/create_feature.csv). And here is my policy gradient [pg](https://github.com/cmcai0104/Assert_Portfolio_Policy/blob/master/PolicyGradient.py).  
  > **username_0**: I want to train a policy by reinforcement learning to improve or exceed the traditional assets portfolio policy. Which use the rolling quantile of PE(the Price Earnings Ratio) of Stocks indexs as the target ratio monthly. The PE is one column in my data. My initial idea was naive. I want to use multiple dense layers to get the actions, and then get the evaluation function(means the probability of this action) by multiple dense layers after actions. But the result is totally decided by the initialization and the training process is also slowly. I think lack of explore strategy is the reason.  
  > **username_1**: Could you confirm which policy gradient (PG) method you are attempting to use? PG is not a method by itself, it is the base theory that is used to create PG methods. Examples of PG methods are REINFORCE, Actor-Critic, PPO, DDPG.  
  
  ---
  
  ## 0 answers
  
  