Answers.Body: <p>The <a href="https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_nn.html" rel="nofollow noreferrer">Pytorch docs</a> define a fully connected ReLU network as:</p>
  <pre><code>torch.nn.Sequential(
      torch.nn.Linear(D_in, H),
      torch.nn.ReLU(),
      torch.nn.Linear(H, D_out),
  )
  </code></pre>
  <p>Neural networks are called are made of neurons. Activation functions only help determine which of these neurons to fire up, meaning they have no learnable nodes themselves through which we can back-propagate gradients. A network with no learnable parameters is therefore not a neural net. So neural nets can't be composed of activation functions only.</p>
  <blockquote>
  <p>What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable</p>
  </blockquote>
  <p>Yes, what's given here is not a network that can approximate the said function <span class="math-container">$h(s)$</span>. A two layer ReLU network would resemble:</p>
  <pre><code>   x = nn.ReLU(nn.Linear(d_IN, H))
     x = nn.ReLU(nn.Linear(H, H))
     out = nn.Linear(x, d_OUT)
  </code></pre>
  <p>Another way to see it is that a network must have an input and ouput layer, and optional hidden layers. It's not possible to use an activation funtion as an input layer, because then you'll have <strong>no way of configuring the number of features to represent your input data</strong>. In this context, a <code>ReLU</code> can't represent the features of the observation input <code>s</code>.</p>
  <p>To show activation functions have no learnable nodes in them and that this interpretation</p>
  <blockquote>
  <p><code>h = nn.Sequential([nn.ReLu(), nn.ReLu()])</code></p>
  </blockquote>
  <p>is not what the authors are driving across, here a script that counts the number of parameters in a network.</p>
  <pre><code>
  import torch.nn as nn
  import numpy as np
  
  activation = nn.ReLU
  
  
  def count_params(module):
      return np.sum([np.prod(x.shape) for x in module.parameters()])
  
  
  one_linear = nn.Sequential(nn.Linear(32, 10), nn.Linear(10, 1))
  linear_act = nn.Sequential(nn.Linear(32, 10), activation(), nn.Linear(10, 1))
  act_only = nn.Sequential(activation(), activation())
  
  t_lin = count_params(linear_act)
  lin = count_params(one_linear)
  act = count_params(act_only)
  
  print(f'Linear only: {lin}, Linear + Activation: {t_lin},' +
          f'Activation only: {act}')
  
  
  </code></pre>
  <p><code>[Out]: Linear only: 341, Linear + Activation: 341, Activation only: 0.0</code></p>
  <p>The activation-function-only module has zero learnable parameters. Likewise, an activation function adds no parameters to the fully connected layers.</p>
  <p><strong>Update: Links to implementations</strong></p>
  <p>To confirm this answer's interpretation is correct, here are links to GAIL and GAN-GCL example implementations</p>
  <ol>
  <li>GAIL : <a href="https://github.com/justinjfu/inverse_rl/blob/9609933389459a3a54f5c01d652114ada90fa1b3/inverse_rl/models/imitation_learning.py#L222" rel="nofollow noreferrer">discriminator prediction</a> (Forward call), <a href="https://github.com/justinjfu/inverse_rl/blob/9609933389459a3a54f5c01d652114ada90fa1b3/inverse_rl/models/architectures.py#L15" rel="nofollow noreferrer">discriminator architecture</a> (The ReLU net):</li>
  <li>GAN-GCL : <a href="https://github.com/justinjfu/inverse_rl/blob/9609933389459a3a54f5c01d652114ada90fa1b3/inverse_rl/models/imitation_learning.py#L415" rel="nofollow noreferrer">discriminator prediction</a>, <a href="https://github.com/justinjfu/inverse_rl/blob/9609933389459a3a54f5c01d652114ada90fa1b3/inverse_rl/models/architectures.py#L29" rel="nofollow noreferrer">discriminator architecture</a>:</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-12T05:57:21.627Z
Answers.LastActivityDate: 2020-11-09T15:43:49.723Z
Answers.LastEditDate: 2020-11-09T15:43:49.723Z
Answers.UserDisplayName: mugoh
Body: <p>Inverse Reinforcement Learning based on <a href="https://arxiv.org/abs/1606.03476" rel="nofollow noreferrer">GAIL</a> and <a href="https://arxiv.org/abs/1603.00448" rel="nofollow noreferrer">GAN-Guided Cost Learning(GAN-GCL)</a>, uses a discriminator to classify between expert demos and policy generated samples.
  <a href="https://arxiv.org/abs/1710.11248" rel="nofollow noreferrer">Adversarial iRL</a>, build upon GAN-GCL, has its discriminator <span class="math-container">$D_{\theta, \phi}$</span> as a function of a state-only reward approximator <span class="math-container">$f_{\theta, \phi}$</span>.</p>
  <p><span class="math-container">$$
  D_{\theta, \phi}\left(s, a, s^{\prime}\right)=\frac{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}}{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}+\pi(a \mid s)},
  $$</span></p>
  <p>where <span class="math-container">$f_{\theta,\phi}$</span> is expressed as:</p>
  <p><span class="math-container">$$f_{\theta,\phi} = g_{\theta} (s) + γh_φ (s\prime ) − h_φ (s).$$</span></p>
  <p>The optimal <span class="math-container">$*g(s)$</span> tries to recover the optimal reward function <span class="math-container">$r^*(s)$</span>. While the <span class="math-container">$h(s)$</span> tries to recover the optimal value funtion <span class="math-container">$V^*(s)$</span>, which makes <span class="math-container">$f_{\theta,\phi}$</span> interpretable as the advantage.</p>
  <p>My question comes from the network architecture used for <span class="math-container">$h(s)$</span> in the original paper.</p>
  <blockquote>
  <p>... we use a <strong>2-layer ReLU network for the shaping term h</strong>. For the policy, we use a <strong>two-layer (32 units) ReLU gaussian policy</strong>.</p>
  </blockquote>
  <p>What is meant by the quoted text in <em>bold</em>, because my interpretation of that text, (shown below) doesn't seem viable</p>
  <pre><code>h = nn.Sequential([nn.ReLu(), nn.ReLu()])
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-09T14:44:19.010Z
Comments.Text: I've seen this [short example from pytorch docs](https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_nn.html). They define a fully-connected ReLU network with one hidden layer as ```nn.Sequential( nn.Linear(in, H), .nn.ReLU(), nn.Linear(H, out))```
Comments.UserDisplayName: mugoh
Tags: <neural-networks><reinforcement-learning><papers><activation-functions><inverse-rl>
Title: Can entire neural networks be composed of only activation functions?
UserDisplayName: mugoh
ThreadText: # Can entire neural networks be composed of only activation functions?
  
  (Asked by: username_0 on 2020-09-06)
  
  Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples.
  Adversarial iRL, build upon GAN-GCL, has its discriminator $D_{\theta, \phi}$ as a function of a state-only reward approximator $f_{\theta, \phi}$.
  $$
  D_{\theta, \phi}\left(s, a, s^{\prime}\right)=\frac{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}}{\exp \left\{f_{\theta, \phi}\left(s, a, s^{\prime}\right)\right\}+\pi(a \mid s)},
  $$
  where $f_{\theta,\phi}$ is expressed as:
  $$f_{\theta,\phi} = g_{\theta} (s) + γh_φ (s\prime ) − h_φ (s).$$
  The optimal $*g(s)$ tries to recover the optimal reward function $r^*(s)$. While the $h(s)$ tries to recover the optimal value funtion $V^*(s)$, which makes $f_{\theta,\phi}$ interpretable as the advantage.
  My question comes from the network architecture used for $h(s)$ in the original paper.
  
  ... we use a 2-layer ReLU network for the shaping term h. For the policy, we use a two-layer (32 units) ReLU gaussian policy.
  
  What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable
  h = nn.Sequential([nn.ReLu(), nn.ReLu()])
  
  **Tags:** neural-networks, reinforcement-learning, papers, activation-functions, inverse-rl
  > **Comments:**
  > **username_0**: I've seen this [short example from pytorch docs](https://pytorch.org/tutorials/beginner/examples_nn/two_layer_net_nn.html). They define a fully-connected ReLU network with one hidden layer as ```nn.Sequential( nn.Linear(in, H), .nn.ReLU(), nn.Linear(H, out))```  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-09-12. Score: 1
  
  The Pytorch docs define a fully connected ReLU network as:
  torch.nn.Sequential(
      torch.nn.Linear(D_in, H),
      torch.nn.ReLU(),
      torch.nn.Linear(H, D_out),
  )
  
  Neural networks are called are made of neurons. Activation functions only help determine which of these neurons to fire up, meaning they have no learnable nodes themselves through which we can back-propagate gradients. A network with no learnable parameters is therefore not a neural net. So neural nets can't be composed of activation functions only.
  
  What is meant by the quoted text in bold, because my interpretation of that text, (shown below) doesn't seem viable
  
  Yes, what's given here is not a network that can approximate the said function $h(s)$. A two layer ReLU network would resemble:
     x = nn.ReLU(nn.Linear(d_IN, H))
     x = nn.ReLU(nn.Linear(H, H))
     out = nn.Linear(x, d_OUT)
  
  Another way to see it is that a network must have an input and ouput layer, and optional hidden layers. It's not possible to use an activation funtion as an input layer, because then you'll have no way of configuring the number of features to represent your input data. In this context, a ReLU can't represent the features of the observation input s.
  To show activation functions have no learnable nodes in them and that this interpretation
  
  h = nn.Sequential([nn.ReLu(), nn.ReLu()])
  
  is not what the authors are driving across, here a script that counts the number of parameters in a network.
  
  import torch.nn as nn
  import numpy as np
  
  activation = nn.ReLU
  
  
  def count_params(module):
      return np.sum([np.prod(x.shape) for x in module.parameters()])
  
  
  one_linear = nn.Sequential(nn.Linear(32, 10), nn.Linear(10, 1))
  linear_act = nn.Sequential(nn.Linear(32, 10), activation(), nn.Linear(10, 1))
  act_only = nn.Sequential(activation(), activation())
  
  t_lin = count_params(linear_act)
  lin = count_params(one_linear)
  act = count_params(act_only)
  
  print(f'Linear only: {lin}, Linear + Activation: {t_lin},' +
          f'Activation only: {act}')
  
  
  
  [Out]: Linear only: 341, Linear + Activation: 341, Activation only: 0.0
  The activation-function-only module has zero learnable parameters. Likewise, an activation function adds no parameters to the fully connected layers.
  Update: Links to implementations
  To confirm this answer's interpretation is correct, here are links to GAIL and GAN-GCL example implementations
  
  GAIL : discriminator prediction (Forward call), discriminator architecture (The ReLU net):
  GAN-GCL : discriminator prediction, discriminator architecture:
  
  
  ---
  
  