Body: <p>I have built an Actor Critic model, where we have a Quantile Critic.
  The aim of the task is to tell what is the optimal portfolio choice of an agent based on quantiles.
  The model set up: We have continuous case so I am using Gaussian policy ( usual set up, entropy regularization etc) and I have simple Quantile Critic model. The  kernel_initializer is HeNormal().Also, I have a Flatten layer before the output layer.</p>
  <p>Why is the loss function for the critic decreasing and the gradient increasing over the episodes?</p>
  <p>Also, the model is too volatile in the testing data.</p>
  <p><a href="https://i.sstatic.net/4ozP2.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4ozP2.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/Indg8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Indg8.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-08T01:51:29.673Z
Comments.Text: Can you please provide a link to this "actor-critic model with quantile critic"? Is this something that you came up with or something that exists in the literature? If the latter, please, provide a link to a research or book that describes it. It might also be a good idea to provide more info about what you mean by "too volatile in the testing data", even if you provide the images.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-10T19:21:53.340Z
Comments.Text: Thank you for answering. The model is mine, I came up with. I cannot really share because of my job. The most important part is the increasing gradient, if you could help me what should i check and explain why it can happen, that would be awesome. What can be the explanation? Isnt is weird that the loss is decreasing parallel? There are some structural breaks in the data, that is why there are some peaks in the loss.
  As for the volatility part: I run many testing episodes and got different result. But this is expected as we have continuous problem with gaussian policy.
Comments.UserDisplayName: dragonforce
Tags: <reinforcement-learning><gradient><advantage-actor-critic><portfolio-optimization>
Title: Why is the loss function for the critic decreasing and the gradient increasing over the episodes?
UserDisplayName: dragonforce
ThreadText: # Why is the loss function for the critic decreasing and the gradient increasing over the episodes?
  
  (Asked by: username_0 on 2024-01-30)
  
  I have built an Actor Critic model, where we have a Quantile Critic.
  The aim of the task is to tell what is the optimal portfolio choice of an agent based on quantiles.
  The model set up: We have continuous case so I am using Gaussian policy ( usual set up, entropy regularization etc) and I have simple Quantile Critic model. The kernel\_initializer is HeNormal().Also, I have a Flatten layer before the output layer.
  
  Why is the loss function for the critic decreasing and the gradient increasing over the episodes?
  
  Also, the model is too volatile in the testing data.
  
  [![enter image description here](https://i.sstatic.net/4ozP2.png)](https://i.sstatic.net/4ozP2.png)
  
  [![enter image description here](https://i.sstatic.net/Indg8.png)](https://i.sstatic.net/Indg8.png)
  
  **Tags:** reinforcement-learning, gradient, advantage-actor-critic, portfolio-optimization
  
  > **Comments:**
  > **username_1**: Can you please provide a link to this "actor-critic model with quantile critic"? Is this something that you came up with or something that exists in the literature? If the latter, please, provide a link to a research or book that describes it. It might also be a good idea to provide more info about what you mean by "too volatile in the testing data", even if you provide the images.  
  > **username_0**: Thank you for answering. The model is mine, I came up with. I cannot really share because of my job. The most important part is the increasing gradient, if you could help me what should i check and explain why it can happen, that would be awesome. What can be the explanation? Isnt is weird that the loss is decreasing parallel? There are some structural breaks in the data, that is why there are some peaks in the loss. As for the volatility part: I run many testing episodes and got different result. But this is expected as we have continuous problem with gaussian policy.  
  
  ---
  
  ## 0 answers
  
  