Answers.Body: <p>Some investigation about the memory dict: The current type is <code>latest</code>, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
  Also, <code>include_next_state</code> means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to <code>False</code>, otherwise an error is raised.
  Your interpretation of <code>capacity</code> looks OK.</p>
  
  <p>About the update mode spec dict, your current settings mean that : every 10 (<code>frequency</code>) episodes (<code>unit</code>), you pull a batch of 10 (<code>batch_size</code>) episodes (<code>unit</code>) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the <code>unit</code> defines both the unit of the optimization frequency and the type of the object fetched from the memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-27T10:12:13.720Z
Answers.LastActivityDate: 2018-08-27T13:24:07.380Z
Answers.LastEditDate: 2018-08-27T13:24:07.380Z
Answers.UserDisplayName: 16Aghnar
Body: <p>I am using the PPO algorithm implemented by tensorforce: <a href="https://github.com/reinforceio/tensorforce" rel="nofollow noreferrer">https://github.com/reinforceio/tensorforce</a> . It works great and I am very happy with the results.</p>
  
  <p>However, I notice that there are many metaparameters available to give to the PPO algorithm:</p>
  
  <pre><code> # the tensorforce agent configuration ------------------------------------------
      network_spec = [
          dict(type='dense', size=256),
          dict(type='dense', size=256),
      ]
  
      agent = PPOAgent(
          states=environment.states,
          actions=environment.actions,
          network=network_spec,
          # Agent
          states_preprocessing=None,
          actions_exploration=None,
          reward_preprocessing=None,
          # MemoryModel
          update_mode=dict(
              unit='episodes',
              # 10 episodes per update
              batch_size=10,
              # Every 10 episodes
              frequency=10
          ),
          memory=dict(
              type='latest',
              include_next_states=False,
              capacity=200000
          ),
          # DistributionModel
          distributions=None,
          entropy_regularization=0.01,
          # PGModel
          baseline_mode='states',
          baseline=dict(
              type='mlp',
              sizes=[32, 32]
          ),
          baseline_optimizer=dict(
              type='multi_step',
              optimizer=dict(
                  type='adam',
                  learning_rate=1e-3
              ),
              num_steps=5
          ),
          gae_lambda=0.97,
          # PGLRModel
          likelihood_ratio_clipping=0.2,
          # PPOAgent
          step_optimizer=dict(
              type='adam',
              learning_rate=1e-3
          ),
          subsampling_fraction=0.2,
          optimization_steps=25,
          execution=dict(
              type='single',
              session_config=None,
              distributed_spec=None
          )
      )
  </code></pre>
  
  <p>So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?</p>
  
  <p>So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?</p>
  
  <ul>
  <li><p>effect of <strong>network_spec</strong>: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.</p></li>
  <li><p>effect of the parameters in <strong>update_mode</strong>: this is how often the network updates are performed.</p>
  
  <ul>
  <li><p><em>batch_size</em> is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?</p></li>
  <li><p><em>frequency</em> is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?</p></li>
  <li><p><em>unit</em>: no idea what this does</p></li>
  </ul></li>
  <li><p><strong>memory</strong>: this is the replay memory buffer.</p>
  
  <ul>
  <li><p><em>type</em>: not sure what this does or how it works.</p></li>
  <li><p><em>include_next_states</em>: not sure what this does or how it works</p></li>
  <li><p><em>capacity</em>: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.</p></li>
  </ul></li>
  <li><p><strong>DistributionMode</strong>: guess this is the model for the distribution of the controls? No idea what the parameters there do.</p></li>
  <li><p><strong>PGModel</strong>: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.</p></li>
  <li><p><strong>PGLRModel</strong>: idem, no idea what all these parameters do / if they should be tweaked.</p></li>
  <li><p><strong>PPOAgend</strong>: idem, no idea what all these parameters do / if they should be tweaked.</p></li>
  </ul>
  
  <p><strong>Summary</strong></p>
  
  <p>So in summary, would be great to get some help about:</p>
  
  <ul>
  <li>Which parameters should be tweaked</li>
  <li>How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?</li>
  </ul>
  
Tags: <deep-learning><reinforcement-learning>
Title: Tuning of PPO metaparameters: a high level overview of what each parameter does
UserDisplayName: Zorglub29
ThreadText: # Tuning of PPO metaparameters: a high level overview of what each parameter does
  
  (Asked by: username_0 on 2018-08-27)
  
  I am using the PPO algorithm implemented by tensorforce: <https://github.com/reinforceio/tensorforce> . It works great and I am very happy with the results.
  
  However, I notice that there are many metaparameters available to give to the PPO algorithm:
  
  ```
   # the tensorforce agent configuration ------------------------------------------
      network_spec = [
          dict(type='dense', size=256),
          dict(type='dense', size=256),
      ]
  
      agent = PPOAgent(
          states=environment.states,
          actions=environment.actions,
          network=network_spec,
          # Agent
          states_preprocessing=None,
          actions_exploration=None,
          reward_preprocessing=None,
          # MemoryModel
          update_mode=dict(
              unit='episodes',
              # 10 episodes per update
              batch_size=10,
              # Every 10 episodes
              frequency=10
          ),
          memory=dict(
              type='latest',
              include_next_states=False,
              capacity=200000
          ),
          # DistributionModel
          distributions=None,
          entropy_regularization=0.01,
          # PGModel
          baseline_mode='states',
          baseline=dict(
              type='mlp',
              sizes=[32, 32]
          ),
          baseline_optimizer=dict(
              type='multi_step',
              optimizer=dict(
                  type='adam',
                  learning_rate=1e-3
              ),
              num_steps=5
          ),
          gae_lambda=0.97,
          # PGLRModel
          likelihood_ratio_clipping=0.2,
          # PPOAgent
          step_optimizer=dict(
              type='adam',
              learning_rate=1e-3
          ),
          subsampling_fraction=0.2,
          optimization_steps=25,
          execution=dict(
              type='single',
              session_config=None,
              distributed_spec=None
          )
      )
  
  ```
  
  So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?
  
  So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?
  
  * effect of **network\_spec**: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.
  * effect of the parameters in **update\_mode**: this is how often the network updates are performed.
  
    + *batch\_size* is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?
    + *frequency* is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?
    + *unit*: no idea what this does
  * **memory**: this is the replay memory buffer.
  
    + *type*: not sure what this does or how it works.
    + *include\_next\_states*: not sure what this does or how it works
    + *capacity*: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.
  * **DistributionMode**: guess this is the model for the distribution of the controls? No idea what the parameters there do.
  * **PGModel**: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.
  * **PGLRModel**: idem, no idea what all these parameters do / if they should be tweaked.
  * **PPOAgend**: idem, no idea what all these parameters do / if they should be tweaked.
  
  **Summary**
  
  So in summary, would be great to get some help about:
  
  * Which parameters should be tweaked
  * How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?
  
  **Tags:** deep-learning, reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-27. Score: 1
  
  Some investigation about the memory dict: The current type is `latest`, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
  Also, `include_next_state` means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to `False`, otherwise an error is raised.
  Your interpretation of `capacity` looks OK.
  
  About the update mode spec dict, your current settings mean that : every 10 (`frequency`) episodes (`unit`), you pull a batch of 10 (`batch_size`) episodes (`unit`) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the `unit` defines both the unit of the optimization frequency and the type of the object fetched from the memory.
  
  
  ---
  
  