Answers.Body: <p>To compare hyperparameters, IMO go ahead with random search. But stop each run after a fixed amount of compute time. (Or a fixed number of steps or episodes, as a proxy for compute time. But keep in mind that some hyperparams may affect the compute cost of a step or episode.)</p>
  <p>With a clever rule to detect when a run has stopped to make progress, you will always wonder whether there would have been a sudden break-through later on. Then you have to justify or debug your stopping criteria, instead of just comparing the runs.</p>
  <p>If you care more about finding a good solution (instead of finding the optimal hyperparameters), then IMO you should use a stopping criterion that compares between different in-progress runs. Common choices for this are ASHA and PBT (population-based training). Reinforcement learning is often very unstable, and stopping bad runs has the additional benefit of filtering out the lucky initializations, which may be more important than finding the exact optimium of the hyperparameters.</p>
  <p>This has the same drawback as above: you can no longer draw any simple conclusion about the hyperparameters. You will never know if one of the &quot;bad&quot; configurations was just a slow starter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-19T16:13:55.717Z
Answers.Text: I  got a bit confused with your anwser. My goal is to find the best solution by random search a "search space of hyperparameters", hence i think that is good idea to stop the run when the agent probably found the local minimum to that solution, so i include the zero loss and the stabilized reward on the stopping criteria.
Answers.UserDisplayName: Vitor Martins
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-19T11:48:59.550Z
Answers.LastActivityDate: 2024-09-19T11:48:59.550Z
Answers.UserDisplayName: maxy
Body: <p>Hello AI Stack people,</p>
  <p>I'm in doubt to when i should stop my DQN agent train.</p>
  <p>I'm traning a DQN agent and i will use a hyperparameter optimization method (probabily random search). So, i need a function that verify if the train iteration has ended, but as i'm trying to find the optimal solution, i want to get the local optimum performance of that train iteration, as that, i think the best approach would be to only end the train when the loss comes to zero, a minimum objective performance is reached and the episode reward has stabilized, all these conditions, for at least some X episodes.</p>
  <p>What do you think about my approach ?</p>
  
Tags: <reinforcement-learning><deep-rl><training><dqn><hyperparameter-optimization>
Title: When to stop a DQN agent train?
UserDisplayName: Vitor Martins
ThreadText: # When to stop a DQN agent train?
  
  (Asked by: username_0 on 2024-09-19)
  
  Hello AI Stack people,
  
  I'm in doubt to when i should stop my DQN agent train.
  
  I'm traning a DQN agent and i will use a hyperparameter optimization method (probabily random search). So, i need a function that verify if the train iteration has ended, but as i'm trying to find the optimal solution, i want to get the local optimum performance of that train iteration, as that, i think the best approach would be to only end the train when the loss comes to zero, a minimum objective performance is reached and the episode reward has stabilized, all these conditions, for at least some X episodes.
  
  What do you think about my approach ?
  
  **Tags:** reinforcement-learning, deep-rl, training, dqn, hyperparameter-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-19. Score: 1
  
  To compare hyperparameters, IMO go ahead with random search. But stop each run after a fixed amount of compute time. (Or a fixed number of steps or episodes, as a proxy for compute time. But keep in mind that some hyperparams may affect the compute cost of a step or episode.)
  
  With a clever rule to detect when a run has stopped to make progress, you will always wonder whether there would have been a sudden break-through later on. Then you have to justify or debug your stopping criteria, instead of just comparing the runs.
  
  If you care more about finding a good solution (instead of finding the optimal hyperparameters), then IMO you should use a stopping criterion that compares between different in-progress runs. Common choices for this are ASHA and PBT (population-based training). Reinforcement learning is often very unstable, and stopping bad runs has the additional benefit of filtering out the lucky initializations, which may be more important than finding the exact optimium of the hyperparameters.
  
  This has the same drawback as above: you can no longer draw any simple conclusion about the hyperparameters. You will never know if one of the "bad" configurations was just a slow starter.
  
  > **Comments:**
  > **username_0**: I  got a bit confused with your anwser. My goal is to find the best solution by random search a "search space of hyperparameters", hence i think that is good idea to stop the run when the agent probably found the local minimum to that solution, so i include the zero loss and the stabilized reward on the stopping criteria.  
  
  ---
  
  