Answers.Body: <p>The short answer is yes, log_softmax + nll_loss will work better. </p>
  
  <p>I don’t know the implementation details under the hood in PyTorch, but see the screenshot below from the <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noreferrer">documentation</a>:</p>
  
  <p><a href="https://i.sstatic.net/BixQf.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BixQf.jpg" alt="img"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T03:54:36.240Z
Answers.Text: Yes, I know that `log_softmax` + `nll_loss` will work better, but I want to know why...
Answers.UserDisplayName: user1024
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T04:33:31.750Z
Answers.Text: @user1024 this seems like a question for the development team since it could depend heavily on their implementation. But you mentioned you used softmax + nll_loss together. Note that since log is monotonically increasing, maximizing the log of the probability with respect to theta is the same as maximizing the probability with respect to theta. That is they both have the same argmax. But log likelihood expects log probabilities, this log_softmax is used. Also note that log(a/b) = log(a) - log(b). But since b is the same for all classes, it’s (probably) omitted in their implementation.
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T04:34:20.167Z
Answers.Text: @user1024 omitted because it wouldn’t affect the maximization, so I assume it would make it faster and more numerically stable. But I also assume they have further optimized the routine implementation in ways that aren’t as obvious simply for speed and stability performance.
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T04:36:49.223Z
Answers.Text: @user1024 and assuming that we are using a Gaussian prior, then maximizing the likelihood requires us to take an exponentiation, but taking the log cancels out the exponentiation, which would also make it faster and more stable.
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-04T03:21:37.033Z
Answers.Text: Thank you @Hanzy, your comments helped explain the advantages of `log_softmax` in speed and numerical properties. As a beginner in AI, I have allodoxaphobia to determine so many super-parameters.
Answers.UserDisplayName: user1024
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-01T02:45:51.337Z
Answers.LastActivityDate: 2019-05-01T02:45:51.337Z
Answers.UserDisplayName: Hanzy
Body: <p>Previously I have learned that the <code>softmax</code> as the output layer coupled with the <code>log-likelihood cost function</code> (the same as the the <code>nll_loss</code> in pytorch) can solve <a href="http://neuralnetworksanddeeplearning.com/chap3.html#softmax" rel="noreferrer">the learning slowdown problem</a>.</p>
  
  <p>However, while I am learning the pytorch mnist tutorial, I'm confused that why the combination of the <code>log_softmax</code> as the output layer and the <code>nll_loss</code>(the negative log likelihood loss) as the loss function was used (<a href="https://github.com/pytorch/examples/blob/master/mnist/main.py#L26" rel="noreferrer">L26</a> and <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py#L34" rel="noreferrer">L34</a>).</p>
  
  <p>I found that when <code>log_softmax</code>+<code>nll_loss</code> was used, the test accuracy was 99%, while when <code>softmax</code>+<code>nll_loss</code> was used, the test accuracy was 97%.</p>
  
  <p>I'm confused that what's the advantage of <code>log_softmax</code> over <code>softmax</code>? How can we explain the performance gap between them? Is <code>log_softmax</code>+<code>nll_loss</code> always better than <code>softmax</code>+<code>nll_loss</code>?</p>
  
Tags: <objective-functions><activation-functions>
Title: What's the advantage of log_softmax over softmax?
UserDisplayName: user1024
ThreadText: # What's the advantage of log_softmax over softmax?
  
  (Asked by: username_0 on 2019-04-30)
  
  Previously I have learned that the `softmax` as the output layer coupled with the `log-likelihood cost function` (the same as the the `nll_loss` in pytorch) can solve [the learning slowdown problem](http://neuralnetworksanddeeplearning.com/chap3.html#softmax).
  
  However, while I am learning the pytorch mnist tutorial, I'm confused that why the combination of the `log_softmax` as the output layer and the `nll_loss`(the negative log likelihood loss) as the loss function was used ([L26](https://github.com/pytorch/examples/blob/master/mnist/main.py#L26) and [L34](https://github.com/pytorch/examples/blob/master/mnist/main.py#L34)).
  
  I found that when `log_softmax`+`nll_loss` was used, the test accuracy was 99%, while when `softmax`+`nll_loss` was used, the test accuracy was 97%.
  
  I'm confused that what's the advantage of `log_softmax` over `softmax`? How can we explain the performance gap between them? Is `log_softmax`+`nll_loss` always better than `softmax`+`nll_loss`?
  
  **Tags:** objective-functions, activation-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-01. Score: 2
  
  The short answer is yes, log\_softmax + nll\_loss will work better.
  
  I don’t know the implementation details under the hood in PyTorch, but see the screenshot below from the [documentation](https://pytorch.org/docs/stable/nn.html):
  
  [![img](https://i.sstatic.net/BixQf.jpg)](https://i.sstatic.net/BixQf.jpg)
  
  > **Comments:**
  > **username_0**: Yes, I know that `log_softmax` + `nll_loss` will work better, but I want to know why...  
  > **username_1**: @user1024 this seems like a question for the development team since it could depend heavily on their implementation. But you mentioned you used softmax + nll_loss together. Note that since log is monotonically increasing, maximizing the log of the probability with respect to theta is the same as maximizing the probability with respect to theta. That is they both have the same argmax. But log likelihood expects log probabilities, this log_softmax is used. Also note that log(a/b) = log(a) - log(b). But since b is the same for all classes, it’s (probably) omitted in their implementation.  
  > **username_1**: @user1024 omitted because it wouldn’t affect the maximization, so I assume it would make it faster and more numerically stable. But I also assume they have further optimized the routine implementation in ways that aren’t as obvious simply for speed and stability performance.  
  > **username_1**: @user1024 and assuming that we are using a Gaussian prior, then maximizing the likelihood requires us to take an exponentiation, but taking the log cancels out the exponentiation, which would also make it faster and more stable.  
  > **username_0**: Thank you @Hanzy, your comments helped explain the advantages of `log_softmax` in speed and numerical properties. As a beginner in AI, I have allodoxaphobia to determine so many super-parameters.  
  
  ---
  
  