Answers.Body: <p>I suggest you should try fixing the seeds. Here's the code how to do it, if you are using <code>torch</code> and the environment is inherited from <a href="https://gymnasium.farama.org/" rel="nofollow noreferrer">gym</a></p>
  <pre><code>import torch
  import random
  import numpy as np
  
  env = import gymnasium as gym
  env = gym.make(&quot;LunarLander-v2&quot;)
  
  def set_global_seed(env, seed=42):
      random.seed(seed)
      os.environ[&quot;PYTHONHASHSEED&quot;] = str(seed)
      np.random.seed(seed)
      torch.manual_seed(seed)
      torch.cuda.manual_seed(seed)
      torch.backends.cudnn.deterministic = True
      torch.backends.cudnn.benchmark = True
      env.seed(seed)
      env.action_space.seed(seed)
  
  set_global_seed(env)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T16:44:18.620Z
Answers.Text: In that case should I experiment with various seeds? In some train session I got good results. Or should the whole process be agnostic of seed i.e. set the seed once and not revisit seed setting ever again. I'm asking this because what if the seed leads to a poor testing result.
Answers.UserDisplayName: ad124j2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T16:55:13.203Z
Answers.Text: The seed will always affect the result. [This paper](https://arxiv.org/abs/2109.08203) experiments with 10^4 seeds and finds it true. However, I recommend you to select about 5 seeds, e.g., 0, 1, 2, 3, 4, then tune the hyperparameters based on those seeds and never touch again. You can do 10 if you are unsure about it
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T17:04:41.417Z
Answers.Text: What if I want to evaluate my architecture? I was thinking of training the agent multiple times and calculate average change which will give me a good idea of expected value of the change. What do you think of this approach?
Answers.UserDisplayName: ad124j2
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T17:24:57.207Z
Answers.Text: That would be the same thing as I mentioned earlier. Fix the seed, do anything you want
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-26T16:30:43.493Z
Answers.LastActivityDate: 2023-02-26T16:30:43.493Z
Answers.UserDisplayName: Minh-Long Luu
Answers.Body: <p>The problem is not that the algorithm is getting inconsistent results and you need to nail it down to one of the &quot;good&quot; strategies.  It is instead that the algorithm is getting inconsistent results and you need to figure out how to get consistently good results.  Depending on how you've tested the good strategy the model could just be over or under-fit and will perform poorly in production.</p>
  <p>If there is a good solution, on a long enough timescale value estimation should converge, no matter the initialization.  In supervised learning we turn to cross validation to give some indication that a model will do well on unseen data.  It is not enough to pick a well behaving initialization there and the principle holds in RL as well.  DQN was not impressive because the authors found a single initialization that played pong, they found an algorithm and data representation that could play pong.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-22T18:23:08.357Z
Answers.LastActivityDate: 2024-03-22T18:23:08.357Z
Answers.UserDisplayName: foreverska
Body: <p>I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and  tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results on testing data (portfolio change at the end of test period). I think, its happening due random initialisation of NN parameters and solution reaching different local maxima. So, how am I to evaluate the agent if it gives anywhere from negative to positive change on every train?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-26T17:30:07.997Z
Comments.Text: Is the trading bot trained *only* on price data? If so, there is typically no usable pattern in that, and you should expect the bot to do about as well as throwing darts at a board with "buy" and "sell" written on it, even if your training was successful. Although there are other possibilities, including that something is wrong with your implementation or training approach or hyperparameters.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><deep-rl><long-short-term-memory><deep-neural-networks><proximal-policy-optimization>
Title: What should I do, reinforcement learning agent gives different result on every train?
UserDisplayName: ad124j2
ThreadText: # What should I do, reinforcement learning agent gives different result on every train?
  
  (Asked by: username_0 on 2023-02-26)
  
  I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results on testing data (portfolio change at the end of test period). I think, its happening due random initialisation of NN parameters and solution reaching different local maxima. So, how am I to evaluate the agent if it gives anywhere from negative to positive change on every train?
  
  **Tags:** reinforcement-learning, deep-rl, long-short-term-memory, deep-neural-networks, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: Is the trading bot trained *only* on price data? If so, there is typically no usable pattern in that, and you should expect the bot to do about as well as throwing darts at a board with "buy" and "sell" written on it, even if your training was successful. Although there are other possibilities, including that something is wrong with your implementation or training approach or hyperparameters.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-02-26. Score: 0
  
  I suggest you should try fixing the seeds. Here's the code how to do it, if you are using `torch` and the environment is inherited from [gym](https://gymnasium.farama.org/)
  
  ```
  import torch
  import random
  import numpy as np
  
  env = import gymnasium as gym
  env = gym.make("LunarLander-v2")
  
  def set_global_seed(env, seed=42):
      random.seed(seed)
      os.environ["PYTHONHASHSEED"] = str(seed)
      np.random.seed(seed)
      torch.manual_seed(seed)
      torch.cuda.manual_seed(seed)
      torch.backends.cudnn.deterministic = True
      torch.backends.cudnn.benchmark = True
      env.seed(seed)
      env.action_space.seed(seed)
  
  set_global_seed(env)
  
  ```
  
  > **Comments:**
  > **username_0**: In that case should I experiment with various seeds? In some train session I got good results. Or should the whole process be agnostic of seed i.e. set the seed once and not revisit seed setting ever again. I'm asking this because what if the seed leads to a poor testing result.  
  > **username_2**: The seed will always affect the result. [This paper](https://arxiv.org/abs/2109.08203) experiments with 10^4 seeds and finds it true. However, I recommend you to select about 5 seeds, e.g., 0, 1, 2, 3, 4, then tune the hyperparameters based on those seeds and never touch again. You can do 10 if you are unsure about it  
  > **username_0**: What if I want to evaluate my architecture? I was thinking of training the agent multiple times and calculate average change which will give me a good idea of expected value of the change. What do you think of this approach?  
  > **username_2**: That would be the same thing as I mentioned earlier. Fix the seed, do anything you want  
  
  ---
  
  ### Answer by username_3 on 2024-03-22. Score: 0
  
  The problem is not that the algorithm is getting inconsistent results and you need to nail it down to one of the "good" strategies. It is instead that the algorithm is getting inconsistent results and you need to figure out how to get consistently good results. Depending on how you've tested the good strategy the model could just be over or under-fit and will perform poorly in production.
  
  If there is a good solution, on a long enough timescale value estimation should converge, no matter the initialization. In supervised learning we turn to cross validation to give some indication that a model will do well on unseen data. It is not enough to pick a well behaving initialization there and the principle holds in RL as well. DQN was not impressive because the authors found a single initialization that played pong, they found an algorithm and data representation that could play pong.
  
  
  ---
  
  