Answers.Body: <p><code>sign</code> is not continuous and not differentiable. Let's say it is defined as follows:</p>
  <p><span class="math-container">$$
  \text{sing}(a) = \begin{cases}
         +1 &amp; \text{if $a&gt;0$}\\
        -1 &amp; \text{otherwise}
      \end{cases}
  $$</span></p>
  <p>where <span class="math-container">$a = \textbf{w}^T\textbf{x}$</span> is a linear pre-activation function.</p>
  <p>The graph will look as follows:</p>
  <blockquote>
  <p><a href="https://i.sstatic.net/qwSFzm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qwSFzm.png" alt="enter image description here" /></a></p>
  </blockquote>
  <p>It is not continuous and, therefore, is not differentiable everywhere. Even if we take the derivative of such a function, it will be equal to zero, since the function is flat. That is, <code>sign</code> does not tell us how good our predictions are, it only says whether they were correct or not.</p>
  <p>More formally, it does not return a gradient that represents the direction in which we should update the <span class="math-container">$a$</span> wrt <span class="math-container">$\textbf{w}$</span>. Instead, we can only move the line (in 2d) trying to correctly classify as many data points as possible by manually changing each weight. If <span class="math-container">$a$</span> were non-linear, it would be impossible to fit a non-linear function by manually changing the weights, since changes in an individual weight would lead to unpredictable changes in the function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-25T12:47:23.707Z
Answers.LastActivityDate: 2021-06-26T09:02:50.473Z
Answers.LastEditDate: 2021-06-26T09:02:50.473Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I am currently studying the textbook <em>Neural Networks and Deep Learning</em> by Charu C. Aggarwal. Chapter <strong>1.2.1.3 Choice of Activation and Loss Functions</strong> says the following:</p>
  <blockquote>
  <p>The classical activation functions that were used early in the development of neural networks were the sign, sigmoid, and the hyperbolic tangent functions:
  <span class="math-container">$$\Phi(v) = \text{sign}(v) \ \ \text{(sign function)} \\ \Phi(v) = \dfrac{1}{1 + e^{-v}} \ \ \text{(sigmoid function)} \\ \Phi(v) = \dfrac{e^{2v} - 1}{e^{2v} + 1} \ \ \text{(tanh function)}$$</span>
  While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.</p>
  </blockquote>
  <p>I am having trouble understanding this part:</p>
  <blockquote>
  <p>While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.</p>
  </blockquote>
  <p>I've read over this a number of times, but I still don't have a good idea of what it is saying (or at least the point it is trying to make). What is this actually saying? What is the point this is trying to make? Perhaps a more detailed explanation of what this is saying will clarify it for me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:37:10.070Z
Comments.Text: `sign` is not continuous and not differentiable, so we cannot get a derivative of it and hence apply gradient descent optimization. Have you looked at [this answer](https://ai.stackexchange.com/questions/28235/where-does-the-so-called-loss-loss-function-fit-into-the-idea-of-a-percept/28283#28283). Does it answer your question?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:38:18.093Z
Comments.Text: @ArayKarjauv Your answer partially clarifies things, but it doesn't explain how this all relates to the perceptron criterion.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:45:23.553Z
Comments.Text: Since the perceptron becomes nondifferentiable, we cannot compute the error and propagate it back through all the nodes. Instead, there are iterative algorithms to optimize the perceptron (see the last part of the answer).
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:54:53.527Z
Comments.Text: @ArayKarjauv Eh, I don't really see how this addresses the part about the perceptron criterion.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:57:39.667Z
Comments.Text: What do you mean by the "perceptron criterion"?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-25T11:58:04.087Z
Comments.Text: @ArayKarjauv This part: "For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation."
Comments.UserDisplayName: The Pointer
Tags: <objective-functions><activation-functions><perceptron>
Title: An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?
UserDisplayName: The Pointer
ThreadText: # An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?
  
  (Asked by: username_0 on 2021-06-25)
  
  I am currently studying the textbook *Neural Networks and Deep Learning* by Charu C. Aggarwal. Chapter **1.2.1.3 Choice of Activation and Loss Functions** says the following:
  
  > The classical activation functions that were used early in the development of neural networks were the sign, sigmoid, and the hyperbolic tangent functions:
  > $$\Phi(v) = \text{sign}(v) \ \ \text{(sign function)} \\ \Phi(v) = \dfrac{1}{1 + e^{-v}} \ \ \text{(sigmoid function)} \\ \Phi(v) = \dfrac{e^{2v} - 1}{e^{2v} + 1} \ \ \text{(tanh function)}$$
  > While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.
  
  I am having trouble understanding this part:
  
  > While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.
  
  I've read over this a number of times, but I still don't have a good idea of what it is saying (or at least the point it is trying to make). What is this actually saying? What is the point this is trying to make? Perhaps a more detailed explanation of what this is saying will clarify it for me.
  
  **Tags:** objective-functions, activation-functions, perceptron
  
  > **Comments:**
  > **username_1**: `sign` is not continuous and not differentiable, so we cannot get a derivative of it and hence apply gradient descent optimization. Have you looked at [this answer](https://ai.stackexchange.com/questions/28235/where-does-the-so-called-loss-loss-function-fit-into-the-idea-of-a-percept/28283#28283). Does it answer your question?  
  > **username_0**: @username_1 Your answer partially clarifies things, but it doesn't explain how this all relates to the perceptron criterion.  
  > **username_1**: Since the perceptron becomes nondifferentiable, we cannot compute the error and propagate it back through all the nodes. Instead, there are iterative algorithms to optimize the perceptron (see the last part of the answer).  
  > **username_0**: @username_1 Eh, I don't really see how this addresses the part about the perceptron criterion.  
  > **username_1**: What do you mean by the "perceptron criterion"?  
  > **username_0**: @username_1 This part: "For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation."  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-25. Score: 3 | Accepted answer
  
  `sign` is not continuous and not differentiable. Let's say it is defined as follows:
  
  $$
  \text{sing}(a) = \begin{cases}
  +1 & \text{if $a>0$}\\
  -1 & \text{otherwise}
  \end{cases}
  $$
  
  where $a = \textbf{w}^T\textbf{x}$ is a linear pre-activation function.
  
  The graph will look as follows:
  
  > [![enter image description here](https://i.sstatic.net/qwSFzm.png)](https://i.sstatic.net/qwSFzm.png)
  
  It is not continuous and, therefore, is not differentiable everywhere. Even if we take the derivative of such a function, it will be equal to zero, since the function is flat. That is, `sign` does not tell us how good our predictions are, it only says whether they were correct or not.
  
  More formally, it does not return a gradient that represents the direction in which we should update the $a$ wrt $\textbf{w}$. Instead, we can only move the line (in 2d) trying to correctly classify as many data points as possible by manually changing each weight. If $a$ were non-linear, it would be impossible to fit a non-linear function by manually changing the weights, since changes in an individual weight would lead to unpredictable changes in the function.
  
  
  ---
  
  