Answers.Body: <p>I would say it is the nature of data. Generally speaking, you are trying to predict a random sequence, especially if you use the history data as an input and try to get the future value as an output.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-04T12:05:45.507Z
Answers.Text: Yes , I am trying another approach now by extracting more data from the temporal series ( historical prices ) by augmenting the data itself, it shows better results but still unstable.
Answers.UserDisplayName: Zahra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T08:57:18.200Z
Answers.LastActivityDate: 2020-06-16T08:57:18.200Z
Answers.UserDisplayName: oleg.mosalov
Body: <p>I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging, and basically it caused bad reward return as well. It sounds like the agent is not learning at all.</p>
  <p>I already tried to tune some hyperparameters (learning rate for each network + number of hidden layers), but I always get similar results.
  The two plots below represent the losses of my policy and one of the value functions during the last episode of training.</p>
  <p><a href="https://i.sstatic.net/r1xdn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/r1xdn.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/3LtkW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3LtkW.png" alt="enter image description here" /></a></p>
  <p>My question is, <strong>would it be related to the data itself (nature of data) or is it something related to the logic of the code?</strong></p>
  
Tags: <reinforcement-learning><tensorflow><actor-critic-methods><hyperparameter-optimization><soft-actor-critic>
Title: Why is my Soft Actor-Critic's policy and value function losses not converging?
UserDisplayName: Zahra
ThreadText: # Why is my Soft Actor-Critic's policy and value function losses not converging?
  
  (Asked by: username_0 on 2020-06-14)
  
  I'm trying to implement a soft actor-critic algorithm for financial data (stock prices), but I have trouble with losses: no matter what combination of hyper-parameters I enter, they are not converging, and basically it caused bad reward return as well. It sounds like the agent is not learning at all.
  
  I already tried to tune some hyperparameters (learning rate for each network + number of hidden layers), but I always get similar results.
  The two plots below represent the losses of my policy and one of the value functions during the last episode of training.
  
  [![enter image description here](https://i.sstatic.net/r1xdn.png)](https://i.sstatic.net/r1xdn.png)
  
  [![enter image description here](https://i.sstatic.net/3LtkW.png)](https://i.sstatic.net/3LtkW.png)
  
  My question is, **would it be related to the data itself (nature of data) or is it something related to the logic of the code?**
  
  **Tags:** reinforcement-learning, tensorflow, actor-critic-methods, hyperparameter-optimization, soft-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-16. Score: 0 | Accepted answer
  
  I would say it is the nature of data. Generally speaking, you are trying to predict a random sequence, especially if you use the history data as an input and try to get the future value as an output.
  
  > **Comments:**
  > **username_0**: Yes , I am trying another approach now by extracting more data from the temporal series ( historical prices ) by augmenting the data itself, it shows better results but still unstable.  
  
  ---
  
  