Answers.Body: <p>Since we're dealing with real-values variables, it is almost certainly the case that the argument of the function will not be <span class="math-container">$0$</span>.</p>
  <p>If you care strongly about that point, you can just use sub-gradients instead (and we do have sub-gradients for this function, so there is no problem).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-27T17:53:45.423Z
Answers.LastActivityDate: 2020-10-27T17:53:45.423Z
Answers.UserDisplayName: Robby Goetschalckx
Answers.Body: <p><span class="math-container">$\max(-y_i(w x_i), 0)$</span> is not partial derivable respect <span class="math-container">$w$</span> if <span class="math-container">$w x_i=0$</span>.</p>
  <p>Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights.</p>
  <p>Assume <span class="math-container">$y_i = 1$</span> and <span class="math-container">$w x_i &lt; 0$</span> (that is, an error of type &quot;false negative&quot;).</p>
  <p>In this case, function <span class="math-container">$[y_i - \text{sign}(w x_i)]^2 = 4$</span>. Derivative on all interval <span class="math-container">$w x_i &lt; 0$</span> is zero, thus, the learning algorithm has no any way to decide if it is better increase or decrease <span class="math-container">$w$</span>.</p>
  <p>In same case, <span class="math-container">$\max(-y_i(w x_i), 0) = - w x_i$</span>, partial derivative is <span class="math-container">$-x_i$</span>. The learning algorithm knows that it must increase <span class="math-container">$w$</span> value if <span class="math-container">$x_i&gt;0$</span>, decrease otherwise. This is the real reason this loss function is considered more practical than previous one.</p>
  <p>How to solve the problem at <span class="math-container">$w x_i = 0$</span> ? simply, if you increase <span class="math-container">$w$</span> and the result is an exact <span class="math-container">$0$</span>, assign to it a very small value, <span class="math-container">$w=\epsilon$</span>. Similar logic for remainder cases.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-27T20:29:48.823Z
Answers.LastActivityDate: 2020-10-27T21:04:52.813Z
Answers.LastEditDate: 2020-10-27T21:04:52.813Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I'm reading chapter one of the book called <a href="https://dl.uswr.ac.ir/bitstream/Hannan/141305/2/9783319944623.pdf" rel="nofollow noreferrer">Neural Networks and Deep Learning</a> from Aggarwal.</p>
  <p>In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use the sign function for the following loss function: <span class="math-container">$\sum_{i=0}^{N}[y_i - \text{sign}(W * X_i)]^2$</span>, that loss function will NOT be differentiable. Therefore, the book suggests us to use, instead of the sign function in the loss function, the perceptron criterion which will be defined as:</p>
  <p><span class="math-container">$$ L_i = \max(-y_i(W * X_i), 0) $$</span></p>
  <p>The question is: Why is the perceptron criterion function differentiable? Won't we face a discontinuity at zero? Is there anything that I'm missing here?</p>
  
Tags: <optimization><gradient-descent><perceptron>
Title: Why is the perceptron criterion function differentiable?
UserDisplayName: Flávio Mendes
ThreadText: # Why is the perceptron criterion function differentiable?
  
  (Asked by: username_0 on 2020-10-27)
  
  I'm reading chapter one of the book called [Neural Networks and Deep Learning](https://dl.uswr.ac.ir/bitstream/Hannan/141305/2/9783319944623.pdf) from Aggarwal.
  
  In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use the sign function for the following loss function: $\sum\_{i=0}^{N}[y\_i - \text{sign}(W \* X\_i)]^2$, that loss function will NOT be differentiable. Therefore, the book suggests us to use, instead of the sign function in the loss function, the perceptron criterion which will be defined as:
  
  $$ L\_i = \max(-y\_i(W \* X\_i), 0) $$
  
  The question is: Why is the perceptron criterion function differentiable? Won't we face a discontinuity at zero? Is there anything that I'm missing here?
  
  **Tags:** optimization, gradient-descent, perceptron
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-10-27. Score: 7 | Accepted answer
  
  $\max(-y\_i(w x\_i), 0)$ is not partial derivable respect $w$ if $w x\_i=0$.
  
  Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights.
  
  Assume $y\_i = 1$ and $w x\_i < 0$ (that is, an error of type "false negative").
  
  In this case, function $[y\_i - \text{sign}(w x\_i)]^2 = 4$. Derivative on all interval $w x\_i < 0$ is zero, thus, the learning algorithm has no any way to decide if it is better increase or decrease $w$.
  
  In same case, $\max(-y\_i(w x\_i), 0) = - w x\_i$, partial derivative is $-x\_i$. The learning algorithm knows that it must increase $w$ value if $x\_i>0$, decrease otherwise. This is the real reason this loss function is considered more practical than previous one.
  
  How to solve the problem at $w x\_i = 0$ ? simply, if you increase $w$ and the result is an exact $0$, assign to it a very small value, $w=\epsilon$. Similar logic for remainder cases.
  
  
  ---
  
  ### Answer by username_1 on 2020-10-27. Score: 3
  
  Since we're dealing with real-values variables, it is almost certainly the case that the argument of the function will not be $0$.
  
  If you care strongly about that point, you can just use sub-gradients instead (and we do have sub-gradients for this function, so there is no problem).
  
  
  ---
  
  