Answers.Body: <p>In modern ML <a href="https://en.wikipedia.org/wiki/Perceptron#Definition" rel="nofollow noreferrer">Perceptron</a> usually means a type of linear binary classifier introduced by Frank Rosenblatt in 1958.</p>
  <blockquote>
  <p>In the modern sense, the perceptron is an algorithm for learning a <em>binary</em> classifier called a threshold function: a function that maps its input <span class="math-container">${\displaystyle \mathbf {x} }$</span> (a real-valued vector) to an output value <span class="math-container">${\displaystyle f(\mathbf {x} )}$</span> (a single binary value): <span class="math-container">${\displaystyle f(\mathbf {x} )=h(\mathbf {w} \cdot \mathbf {x} +b)}$</span>, where <span class="math-container">${\displaystyle h}$</span> is the Heaviside <em>step-function</em></p>
  </blockquote>
  <p>And of course there's nothing wrong for a single-layer perceptron refers to a more <em>general</em> neural network with <em>real-valued</em> output via your mentioned nonlinear activation function. So you should always judge the almost synonyms definition of either a perceptron or a single layer of perceptron based on their context.</p>
  <p>Finally even with a non-linear activation function, a single-layer perceptron cannot model highly complex non-linear relationships. This limitation is due to the lack of hidden layers which are critical for feature transformation and hierarchical representation of data. For instance, the classic XOR problem (which is not linearly separable) cannot be solved by a single-layer perceptron even with a non-linear activation function.</p>
  <blockquote>
  <p>A single perceptron can learn to classify any half-space. It cannot solve any linearly nonseparable vectors, such as the Boolean exclusive-or problem (the famous &quot;XOR problem&quot;).</p>
  </blockquote>
  <blockquote>
  <p>A perceptron network with one hidden layer can learn to classify any compact subset arbitrarily closely. Similarly, it can also approximate any compactly-supported continuous function arbitrarily closely. This is essentially a special case of the theorems by George Cybenko and Kurt Hornik.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T09:34:08.080Z
Answers.Text: Thanks for the answer. Our learning material was Jurafsky and Martin's book. In [Ch7](https://web.stanford.edu/~jurafsky/slp3/7.pdf), Section 7.2, it is written in bold that a perceptron does not have a non-linear activation function. As you pointed out and also when you google it, people use Heaviside function for activation. Is the Heaviside function a linear function? I don't think it is because you can't come up with $a,b$ such $f(x) = ax + b$ is the same as Heaviside, but perhaps I'm wrong...
Answers.UserDisplayName: Yuirike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T09:37:53.113Z
Answers.Text: Indeed Heaviside step function is nonlinear too strictly speaking, but its output is binary, which was use for classification purpose. While all other usual nonlinear activation functions you mentioned output is continuous real numbers, which can also be used for regression purposes. Hope this clarifies your lingering confusion for this post.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T10:13:55.827Z
Answers.Text: The confusion is still not quite resolved yet. According to your answer, you need a non-linearity to be called a perceptron (if Heaviside is non-linear as well). According to Jurafsky and Martin, you do not. So someone is wrong or some part is still left unclear.
Answers.UserDisplayName: Yuirike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T19:38:13.763Z
Answers.Text: Maybe you shouldn't nickpick for Jurafsky and Martin's choice of words. When people say "non-linear activation functions" in the context of modern neural networks, they often mean smooth continuous non-linearities to produce real-valued numbers like those you mentioned. And strictly speaking, step function is non-linear, so I believe this minor semantic issue caused by their non-strict use of words in language composition should settle now.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-15T09:17:01.333Z
Answers.LastActivityDate: 2024-12-15T09:17:01.333Z
Answers.UserDisplayName: cinch
Body: <p>I couldn't find the exact answer to my question because the discussions tend go somewhere else, so I want to ask:</p>
  <p><strong>What is a single-layer perceptron and what makes it different from 'perceptron'? Are those two synonyms?</strong></p>
  <p>To my understanding: <span class="math-container">$y = W \mathbf{x}$</span> is a single-layer perceptron, with <span class="math-container">$y$</span> being the output layer and the input layer not being counted. There is a single 'linear' transformation.</p>
  <p>However, you can in theory just use an activation function that is 'non-linear'
  <span class="math-container">$$y = \sigma(W\mathbf(x))$$</span></p>
  <p>Where <span class="math-container">$\sigma$</span> could be ReLu, Sigmoid, tanh or some exotic activation function that is not linear at all.</p>
  <p><strong>Can a single layer perceptron use non-linear activation function or is this considered improper?</strong></p>
  <p><strong>If yes, is this model able to 'mimick' a non-linear function?</strong></p>
  <p>To my understanding, the whole goal of neural networks is that we assume there is a function that can solve our problem but we aren't really able to figure it out using math, so we approximate it instead using adjustable parameters. <strong>Is a single layer perceptron unable to ever approximate a non-linear function,  even though a non-linearity is performed on the output?</strong></p>
  
Tags: <neural-networks><definitions><activation-functions><perceptron>
Title: What is the difference between SLP and perceptron and do they require non-linearity or not?
UserDisplayName: Yuirike
ThreadText: # What is the difference between SLP and perceptron and do they require non-linearity or not?
  
  (Asked by: username_0 on 2024-12-14)
  
  I couldn't find the exact answer to my question because the discussions tend go somewhere else, so I want to ask:
  
  **What is a single-layer perceptron and what makes it different from 'perceptron'? Are those two synonyms?**
  
  To my understanding: $y = W \mathbf{x}$ is a single-layer perceptron, with $y$ being the output layer and the input layer not being counted. There is a single 'linear' transformation.
  
  However, you can in theory just use an activation function that is 'non-linear'
  $$y = \sigma(W\mathbf(x))$$
  
  Where $\sigma$ could be ReLu, Sigmoid, tanh or some exotic activation function that is not linear at all.
  
  **Can a single layer perceptron use non-linear activation function or is this considered improper?**
  
  **If yes, is this model able to 'mimick' a non-linear function?**
  
  To my understanding, the whole goal of neural networks is that we assume there is a function that can solve our problem but we aren't really able to figure it out using math, so we approximate it instead using adjustable parameters. **Is a single layer perceptron unable to ever approximate a non-linear function, even though a non-linearity is performed on the output?**
  
  **Tags:** neural-networks, definitions, activation-functions, perceptron
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-15. Score: 1
  
  In modern ML [Perceptron](https://en.wikipedia.org/wiki/Perceptron#Definition) usually means a type of linear binary classifier introduced by Frank Rosenblatt in 1958.
  
  > In the modern sense, the perceptron is an algorithm for learning a *binary* classifier called a threshold function: a function that maps its input ${\displaystyle \mathbf {x} }$ (a real-valued vector) to an output value ${\displaystyle f(\mathbf {x} )}$ (a single binary value): ${\displaystyle f(\mathbf {x} )=h(\mathbf {w} \cdot \mathbf {x} +b)}$, where ${\displaystyle h}$ is the Heaviside *step-function*
  
  And of course there's nothing wrong for a single-layer perceptron refers to a more *general* neural network with *real-valued* output via your mentioned nonlinear activation function. So you should always judge the almost synonyms definition of either a perceptron or a single layer of perceptron based on their context.
  
  Finally even with a non-linear activation function, a single-layer perceptron cannot model highly complex non-linear relationships. This limitation is due to the lack of hidden layers which are critical for feature transformation and hierarchical representation of data. For instance, the classic XOR problem (which is not linearly separable) cannot be solved by a single-layer perceptron even with a non-linear activation function.
  
  > A single perceptron can learn to classify any half-space. It cannot solve any linearly nonseparable vectors, such as the Boolean exclusive-or problem (the famous "XOR problem").
  
  > A perceptron network with one hidden layer can learn to classify any compact subset arbitrarily closely. Similarly, it can also approximate any compactly-supported continuous function arbitrarily closely. This is essentially a special case of the theorems by George Cybenko and Kurt Hornik.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. Our learning material was Jurafsky and Martin's book. In [Ch7](https://web.stanford.edu/~jurafsky/slp3/7.pdf), Section 7.2, it is written in bold that a perceptron does not have a non-linear activation function. As you pointed out and also when you google it, people use Heaviside function for activation. Is the Heaviside function a linear function? I don't think it is because you can't come up with $a,b$ such $f(x) = ax + b$ is the same as Heaviside, but perhaps I'm wrong...  
  > **username_1**: Indeed Heaviside step function is nonlinear too strictly speaking, but its output is binary, which was use for classification purpose. While all other usual nonlinear activation functions you mentioned output is continuous real numbers, which can also be used for regression purposes. Hope this clarifies your lingering confusion for this post.  
  > **username_0**: The confusion is still not quite resolved yet. According to your answer, you need a non-linearity to be called a perceptron (if Heaviside is non-linear as well). According to Jurafsky and Martin, you do not. So someone is wrong or some part is still left unclear.  
  > **username_1**: Maybe you shouldn't nickpick for Jurafsky and Martin's choice of words. When people say "non-linear activation functions" in the context of modern neural networks, they often mean smooth continuous non-linearities to produce real-valued numbers like those you mentioned. And strictly speaking, step function is non-linear, so I believe this minor semantic issue caused by their non-strict use of words in language composition should settle now.  
  
  ---
  
  