Answers.Body: <p>Some investigation about the memory dict: The current type is <code>latest</code>, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
  Also, <code>include_next_state</code> means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to <code>False</code>, otherwise an error is raised.
  Your interpretation of <code>capacity</code> looks OK.</p>
  
  <p>About the update mode spec dict, your current settings mean that : every 10 (<code>frequency</code>) episodes (<code>unit</code>), you pull a batch of 10 (<code>batch_size</code>) episodes (<code>unit</code>) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the <code>unit</code> defines both the unit of the optimization frequency and the type of the object fetched from the memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-27T10:12:13.720Z
Answers.LastActivityDate: 2018-08-27T13:24:07.380Z
Answers.LastEditDate: 2018-08-27T13:24:07.380Z
Answers.UserDisplayName: 16Aghnar
Body: <p>I am using the PPO algorithm implemented by tensorforce: <a href="https://github.com/reinforceio/tensorforce" rel="nofollow noreferrer">https://github.com/reinforceio/tensorforce</a> . It works great and I am very happy with the results.</p>
  
  <p>However, I notice that there are many metaparameters available to give to the PPO algorithm:</p>
  
  <pre><code> # the tensorforce agent configuration ------------------------------------------
      network_spec = [
          dict(type='dense', size=256),
          dict(type='dense', size=256),
      ]
  
      agent = PPOAgent(
          states=environment.states,
          actions=environment.actions,
          network=network_spec,
          # Agent
          states_preprocessing=None,
          actions_exploration=None,
          reward_preprocessing=None,
          # MemoryModel
          update_mode=dict(
              unit='episodes',
              # 10 episodes per update
              batch_size=10,
              # Every 10 episodes
              frequency=10
          ),
          memory=dict(
              type='latest',
              include_next_states=False,
              capacity=200000
          ),
          # DistributionModel
          distributions=None,
          entropy_regularization=0.01,
          # PGModel
          baseline_mode='states',
          baseline=dict(
              type='mlp',
              sizes=[32, 32]
          ),
          baseline_optimizer=dict(
              type='multi_step',
              optimizer=dict(
                  type='adam',
                  learning_rate=1e-3
              ),
              num_steps=5
          ),
          gae_lambda=0.97,
          # PGLRModel
          likelihood_ratio_clipping=0.2,
          # PPOAgent
          step_optimizer=dict(
              type='adam',
              learning_rate=1e-3
          ),
          subsampling_fraction=0.2,
          optimization_steps=25,
          execution=dict(
              type='single',
              session_config=None,
              distributed_spec=None
          )
      )
  </code></pre>
  
  <p>So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?</p>
  
  <p>So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?</p>
  
  <ul>
  <li><p>effect of <strong>network_spec</strong>: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.</p></li>
  <li><p>effect of the parameters in <strong>update_mode</strong>: this is how often the network updates are performed.</p>
  
  <ul>
  <li><p><em>batch_size</em> is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?</p></li>
  <li><p><em>frequency</em> is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?</p></li>
  <li><p><em>unit</em>: no idea what this does</p></li>
  </ul></li>
  <li><p><strong>memory</strong>: this is the replay memory buffer.</p>
  
  <ul>
  <li><p><em>type</em>: not sure what this does or how it works.</p></li>
  <li><p><em>include_next_states</em>: not sure what this does or how it works</p></li>
  <li><p><em>capacity</em>: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.</p></li>
  </ul></li>
  <li><p><strong>DistributionMode</strong>: guess this is the model for the distribution of the controls? No idea what the parameters there do.</p></li>
  <li><p><strong>PGModel</strong>: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.</p></li>
  <li><p><strong>PGLRModel</strong>: idem, no idea what all these parameters do / if they should be tweaked.</p></li>
  <li><p><strong>PPOAgend</strong>: idem, no idea what all these parameters do / if they should be tweaked.</p></li>
  </ul>
  
  <p><strong>Summary</strong></p>
  
  <p>So in summary, would be great to get some help about:</p>
  
  <ul>
  <li>Which parameters should be tweaked</li>
  <li>How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?</li>
  </ul>
  
Tags: <deep-learning><reinforcement-learning>
Title: Tuning of PPO metaparameters: a high level overview of what each parameter does
UserDisplayName: Zorglub29
ThreadText: # Tuning of PPO metaparameters: a high level overview of what each parameter does
  
  (Asked by: username_0 on 2018-08-27)
  
  I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results.
  However, I notice that there are many metaparameters available to give to the PPO algorithm:
   # the tensorforce agent configuration ------------------------------------------
      network_spec = [
          dict(type='dense', size=256),
          dict(type='dense', size=256),
      ]
  
      agent = PPOAgent(
          states=environment.states,
          actions=environment.actions,
          network=network_spec,
          # Agent
          states_preprocessing=None,
          actions_exploration=None,
          reward_preprocessing=None,
          # MemoryModel
          update_mode=dict(
              unit='episodes',
              # 10 episodes per update
              batch_size=10,
              # Every 10 episodes
              frequency=10
          ),
          memory=dict(
              type='latest',
              include_next_states=False,
              capacity=200000
          ),
          # DistributionModel
          distributions=None,
          entropy_regularization=0.01,
          # PGModel
          baseline_mode='states',
          baseline=dict(
              type='mlp',
              sizes=[32, 32]
          ),
          baseline_optimizer=dict(
              type='multi_step',
              optimizer=dict(
                  type='adam',
                  learning_rate=1e-3
              ),
              num_steps=5
          ),
          gae_lambda=0.97,
          # PGLRModel
          likelihood_ratio_clipping=0.2,
          # PPOAgent
          step_optimizer=dict(
              type='adam',
              learning_rate=1e-3
          ),
          subsampling_fraction=0.2,
          optimization_steps=25,
          execution=dict(
              type='single',
              session_config=None,
              distributed_spec=None
          )
      )
  
  So my question is: is there a way to understand, intuitively, the meaning / effect of all these metaparameters and use this intuitive understanding to improve training performance?
  So far I have reached - from a mix of reading the PPO paper and the literature around, and playing with the code - to the following conclusions. Can anybody complete / correct?
  
  effect of network_spec: this is size of the 'main network'. Quite classical: need it big enough to get valuable predictions, not too big either otherwise it is hard to train.
  effect of the parameters in update_mode: this is how often the network updates are performed.
  
  batch_size is how many used for a batch update. Not sure of the effect neither what this exactly means in practice (are all samples taken from only 10 batches of the memory replay)?
  frequency is how often the update is performed. I guess having frequency high would make the training slower but more stable (as sample from more different batches)?
  unit: no idea what this does
  
  memory: this is the replay memory buffer.
  
  type: not sure what this does or how it works.
  include_next_states: not sure what this does or how it works
  capacity: I think this is how many tuples (state, action, reward) are stored. I think this is an important metaparameter. In my experience, if this is too low compared to the number of actions in one episode, the learning is very bad. I guess this is because it must be large enough to store MANY episodes, otherwise the network learns from correlated data - which is bad.
  
  DistributionMode: guess this is the model for the distribution of the controls? No idea what the parameters there do.
  PGModel: No idea what the paramaters there do. Would be interesting to know if some should be tweaked / which ones.
  PGLRModel: idem, no idea what all these parameters do / if they should be tweaked.
  PPOAgend: idem, no idea what all these parameters do / if they should be tweaked.
  
  Summary
  So in summary, would be great to get some help about:
  
  Which parameters should be tweaked
  How should these parameters be tweaked? Is there a 'high level intuition' about how they should be tweaked / in which circumstances?
  
  **Tags:** deep-learning, reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-08-27. Score: 1
  
  Some investigation about the memory dict: The current type is `latest`, which means you're not using a memory replay, but a latest memory. Switching to replay may help.
  Also, `include_next_state` means that you store tuples (state, action, reward, next state). It's not a real parameter though, because in PPO it must be set to `False`, otherwise an error is raised.
  Your interpretation of `capacity` looks OK.
  
  About the update mode spec dict, your current settings mean that : every 10 (`frequency`) episodes (`unit`), you pull a batch of 10 (`batch_size`) episodes (`unit`) from the memory (the pulling method is defined through the memory dict), and you perform an optimization step over this batch. Be aware that the `unit` defines both the unit of the optimization frequency and the type of the object fetched from the memory.
  
  
  ---
  
  