Answers.Body: <p>A neural network layer with no activation function is the same as &quot;linear activation&quot; i.e. <span class="math-container">$f(x) = x$</span></p>
  <p>This is often used for the output layer in regression problems, where a constrained output like that of sigmoid, hyperbolic tangent or ReLU may not be appropriate.</p>
  <p>For an output layer, this is fine, and does not conflict with any theory behind neural networks. It will often be combined with a mean squared error (MSE) loss function that makes the gradient calculation simple at the output layer.</p>
  <p>For hidden layers, skipping the activation function can be a problem, since a purely linear layer in the <em>middle</em> of a multi-layer network is redundant - it could be replaced or even removed with no impact. That is because two directly connected linear layers are functionally equivalent to a single linear layer with different parameters, and every hidden layer consists of a linear component plus an activation function. So even one missing activation function on a hidden layer directly connects two linear sub-components, making one of them redundant.</p>
  <p>In the case of a classifier, some libraries (notably including PyTorch which you are using) require you to have two variants of your neural network - a trainable version without a final sigmoid or softmax layer, and the &quot;full&quot; version which adds sigmoid or softmax on top (usually just a function that calls the training network and add this last activation). This is done to allow you to use more stable gradient calculations that are based on the <em>logits</em>, the values before applying an activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T13:11:14.150Z
Answers.Text: Thanks for this.  Exactly the point I was making in my comment.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T13:35:56.913Z
Answers.Text: While this is correct, it does not answer the question. The problem considered in the example is multiclass classification with cross-entropy loss. As far as I know, it requires softmax activation. Will it work with linear activation?
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T13:51:18.537Z
Answers.Text: @ArayKarjauv Actually yes it will, sort of. It is common in PyTorch to not implement the final softmax, and to use a more stable loss function gradient function on the logits. Then there is also an extra method added to the nn module that runs the softmax when needed. So you will often see an incomplete variant of your network used for training purposes only. I added a para about that
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T14:08:40.153Z
Answers.Text: I see. So the network does have softmax activation for the output layer, which is integrated into [F.cross_entropy](https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy)
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T14:47:19.410Z
Answers.Text: @ArayKarjauv That's not the specific function I recall using, I would use https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss - but I think they are very similar just with different calling conventions, and likely resolve to the same optimised C code underneath
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T12:17:23.930Z
Answers.LastActivityDate: 2021-08-02T13:54:41.487Z
Answers.LastEditDate: 2021-08-02T13:54:41.487Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I'd like to add more information to Neal's answer.</p>
  <p>A network implemented in the example does not include activation for the output, because it will be applied during the training:</p>
  <pre><code># ...
  def training_step(self, batch):
      images, labels = batch 
      out = self(images)
      # apply activation and calculate loss
      loss = F.cross_entropy(out, labels)
      return loss
  # ...
  </code></pre>
  <p>The problem considered in the example is multiclass classification, which is usually solved with softmax activation and cross-entropy loss. <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy" rel="nofollow noreferrer">F.cross_entropy</a> combines <code>log_softmax</code> and <code>nll_loss</code> in a single function, which is numerically more stable as softmax and NLL loss. See this <a href="https://discuss.pytorch.org/t/logsoftmax-vs-softmax/21386/4" rel="nofollow noreferrer">discussion</a> for more details, and here is an <a href="https://jamesmccaffrey.wordpress.com/2020/06/11/pytorch-crossentropyloss-vs-nllloss-cross-entropy-loss-vs-negative-log-likelihood-loss/" rel="nofollow noreferrer">article</a> about different implementations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T16:47:54.110Z
Answers.Text: But why isn't there the activation function on the output layer during calling the forward function? @ArayKarjauv
Answers.UserDisplayName: Omar Zayed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T17:04:33.383Z
Answers.Text: @OmarZayed That is unnecessary. PyTorch allows additional calculations (e.g., mean, normalization, multiplication, etc.) to be done outside of the model class.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T12:43:13.147Z
Answers.Text: So when is adding an activation function necessary in the output layer?
Answers.UserDisplayName: Omar Zayed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T14:05:35.157Z
Answers.Text: @OmarZayed when you use the [negative log-likelihood loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html) alone or calculate it manually (in that case you can also apply softmax manually outside the `forward` method). Please read this [article](https://jamesmccaffrey.wordpress.com/2020/06/11/pytorch-crossentropyloss-vs-nllloss-cross-entropy-loss-vs-negative-log-likelihood-loss/) for more information.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T20:10:06.117Z
Answers.Text: so according to my understanding, the output layer shouldn't have an activation function because the only thing we care about is getting the highest value for some label?
Answers.UserDisplayName: Omar Zayed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T20:27:25.520Z
Answers.Text: @OmarZayed what do you mean with the highest value? You shouldn't apply softmax activation to the output layer, because the `F.cross_entropy` loss function will do it for you.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T14:49:27.720Z
Answers.LastActivityDate: 2021-08-02T17:05:26.123Z
Answers.LastEditDate: 2021-08-02T17:05:26.123Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I have enrolled in a <a href="https://jovian.ai/kukuquack/04-feedforward-nn" rel="nofollow noreferrer">course</a> that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:</p>
  <p><a href="https://i.sstatic.net/njZsxm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/njZsxm.png" alt="enter image description here" /></a></p>
  <p>and here is a PyTorch implementation:</p>
  <pre><code>class MnistModel(nn.Module):
      &quot;&quot;&quot;Feedfoward neural network with 1 hidden layer&quot;&quot;&quot;
      def __init__(self, in_size, hidden_size, out_size):
          super().__init__()
          # hidden layer
          self.linear1 = nn.Linear(in_size, hidden_size)
          # output layer
          self.linear2 = nn.Linear(hidden_size, out_size)
          
      def forward(self, xb):
          # Flatten the image tensors
          xb = xb.view(xb.size(0), -1)
          # Get intermediate outputs using hidden layer
          out = self.linear1(xb)
          # Apply activation function
          out = F.relu(out)
          # Get predictions using output layer
          out = self.linear2(out)
          return out
  </code></pre>
  <p>shouldn't the output layer also have activation functions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T14:55:50.070Z
Comments.Text: All layers (except input) have an activation. If it is not defined explicitly, it will be the linear activation. Depending on your task, you may need the linear activation (e.g., for regression)
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T18:29:05.790Z
Comments.Text: @ArayKarjauv Thank you for replying. In the course I'm doing, there is no activation function on the output layer. Do you know why is that?
Comments.UserDisplayName: Omar Zayed
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T19:25:12.903Z
Comments.Text: can you please provide a piece of additional information or an example?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T20:49:59.997Z
Comments.Text: Well, this isn't strictly true.  For example, I have some regression networks where the output layer is designed to *not* have an activation.  It really depends on what you're doing.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T09:45:44.050Z
Comments.Text: @ArayKarjauv https://jovian.ai/kukuquack/04-feedforward-nn
  this is my notebook, in one of the code cells, there are 2 linear layers and one activation function for the first layer, You will find it in the mnistmodel class, forward function.
Comments.UserDisplayName: Omar Zayed
Tags: <neural-networks><deep-learning><activation-functions>
Title: Does the output layer in a deep neural network need an activation function?
UserDisplayName: Omar Zayed
ThreadText: # Does the output layer in a deep neural network need an activation function?
  
  (Asked by: username_0 on 2021-08-01)
  
  I have enrolled in a course that uses only one hidden layer, and that is the only layer that has activation functions. The model can be visualized as follows:
  
  and here is a PyTorch implementation:
  class MnistModel(nn.Module):
      """Feedfoward neural network with 1 hidden layer"""
      def __init__(self, in_size, hidden_size, out_size):
          super().__init__()
          # hidden layer
          self.linear1 = nn.Linear(in_size, hidden_size)
          # output layer
          self.linear2 = nn.Linear(hidden_size, out_size)
          
      def forward(self, xb):
          # Flatten the image tensors
          xb = xb.view(xb.size(0), -1)
          # Get intermediate outputs using hidden layer
          out = self.linear1(xb)
          # Apply activation function
          out = F.relu(out)
          # Get predictions using output layer
          out = self.linear2(out)
          return out
  
  shouldn't the output layer also have activation functions?
  
  **Tags:** neural-networks, deep-learning, activation-functions
  > **Comments:**
  > **username_1**: All layers (except input) have an activation. If it is not defined explicitly, it will be the linear activation. Depending on your task, you may need the linear activation (e.g., for regression)  
  > **username_0**: @ArayKarjauv Thank you for replying. In the course I'm doing, there is no activation function on the output layer. Do you know why is that?  
  > **username_1**: can you please provide a piece of additional information or an example?  
  > **username_2**: Well, this isn't strictly true.  For example, I have some regression networks where the output layer is designed to *not* have an activation.  It really depends on what you're doing.  
  > **username_0**: @ArayKarjauv https://jovian.ai/kukuquack/04-feedforward-nn this is my notebook, in one of the code cells, there are 2 linear layers and one activation function for the first layer, You will find it in the mnistmodel class, forward function.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-08-02. Score: 2
  
  A neural network layer with no activation function is the same as "linear activation" i.e. $f(x) = x$
  
  This is often used for the output layer in regression problems, where a constrained output like that of sigmoid, hyperbolic tangent or ReLU may not be appropriate.
  
  For an output layer, this is fine, and does not conflict with any theory behind neural networks. It will often be combined with a mean squared error (MSE) loss function that makes the gradient calculation simple at the output layer.
  
  For hidden layers, skipping the activation function can be a problem, since a purely linear layer in the *middle* of a multi-layer network is redundant - it could be replaced or even removed with no impact. That is because two directly connected linear layers are functionally equivalent to a single linear layer with different parameters, and every hidden layer consists of a linear component plus an activation function. So even one missing activation function on a hidden layer directly connects two linear sub-components, making one of them redundant.
  
  In the case of a classifier, some libraries (notably including PyTorch which you are using) require you to have two variants of your neural network - a trainable version without a final sigmoid or softmax layer, and the "full" version which adds sigmoid or softmax on top (usually just a function that calls the training network and add this last activation). This is done to allow you to use more stable gradient calculations that are based on the *logits*, the values before applying an activation function.
  
  > **Comments:**
  > **username_2**: Thanks for this.  Exactly the point I was making in my comment.  
  > **username_1**: While this is correct, it does not answer the question. The problem considered in the example is multiclass classification with cross-entropy loss. As far as I know, it requires softmax activation. Will it work with linear activation?  
  > **username_3**: @ArayKarjauv Actually yes it will, sort of. It is common in PyTorch to not implement the final softmax, and to use a more stable loss function gradient function on the logits. Then there is also an extra method added to the nn module that runs the softmax when needed. So you will often see an incomplete variant of your network used for training purposes only. I added a para about that  
  > **username_1**: I see. So the network does have softmax activation for the output layer, which is integrated into [F.cross_entropy](https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy)  
  > **username_3**: @ArayKarjauv That's not the specific function I recall using, I would use https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss - but I think they are very similar just with different calling conventions, and likely resolve to the same optimised C code underneath  
  
  ---
  
  ### Answer by username_1 on 2021-08-02. Score: 2
  
  I'd like to add more information to Neal's answer.
  A network implemented in the example does not include activation for the output, because it will be applied during the training:
  # ...
  def training_step(self, batch):
      images, labels = batch 
      out = self(images)
      # apply activation and calculate loss
      loss = F.cross_entropy(out, labels)
      return loss
  # ...
  
  The problem considered in the example is multiclass classification, which is usually solved with softmax activation and cross-entropy loss. F.cross_entropy combines log_softmax and nll_loss in a single function, which is numerically more stable as softmax and NLL loss. See this discussion for more details, and here is an article about different implementations.
  
  > **Comments:**
  > **username_0**: But why isn't there the activation function on the output layer during calling the forward function? @ArayKarjauv  
  > **username_1**: @OmarZayed That is unnecessary. PyTorch allows additional calculations (e.g., mean, normalization, multiplication, etc.) to be done outside of the model class.  
  > **username_0**: So when is adding an activation function necessary in the output layer?  
  > **username_1**: @OmarZayed when you use the [negative log-likelihood loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html) alone or calculate it manually (in that case you can also apply softmax manually outside the `forward` method). Please read this [article](https://jamesmccaffrey.wordpress.com/2020/06/11/pytorch-crossentropyloss-vs-nllloss-cross-entropy-loss-vs-negative-log-likelihood-loss/) for more information.  
  > **username_0**: so according to my understanding, the output layer shouldn't have an activation function because the only thing we care about is getting the highest value for some label?  
  > **username_1**: @OmarZayed what do you mean with the highest value? You shouldn't apply softmax activation to the output layer, because the `F.cross_entropy` loss function will do it for you.  
  
  ---
  
  