Answers.Body: <p>A non-linear layer is a matrix multiplication followed by an activation function. A linear layer is just a matrix multiplication.</p>
  <p>You have a matrix multiplication, then another matrix multiplication, then an activation function.</p>
  <p>Two matrix multiplications in series are equivalent to one matrix multiplication, so your network is equivalent to one non-linear layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-25T03:54:35.087Z
Answers.LastActivityDate: 2023-03-25T03:54:35.087Z
Answers.UserDisplayName: Criticize SE actions means ban
Answers.Body: <p>You are speaking about generalized linear models (GLM)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T12:56:08.660Z
Answers.Text: Do the model GLM follow semi linearity? I am new to this. Could you please help to provide detail on how it works?
Answers.UserDisplayName: Prabal Devkota
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T11:01:39.090Z
Answers.LastActivityDate: 2023-03-19T11:01:39.090Z
Answers.UserDisplayName: Ggjj11
Answers.Body: <p>If you have a <code>sigmoid</code> activation in your output layer, and linearities before, you can interpret it as a <a href="https://en.wikipedia.org/wiki/Logistic_regression" rel="nofollow noreferrer">logistic regression</a> model for binary classification (if output dimension is one.)</p>
  <p>Also, in practice, having multiple <em>linear</em> hidden layers is a waste of resources because you can define an equivalent linear layer whose weight matrix is the linear combination of those.</p>
  <p>Moreover, intermediate linear layers can be useful to &quot;adjust&quot; (i.e. expand or decrease) the output dimension of the previous layer (including the input layer.)</p>
  <p>In general, you want to have multiple non-linear layers (i.e. whatever layer with non-linear activation on top) in the middle of your model (i.e. as hidden layers) because the activation of the output layer is often (if not always) problem dependent: e.g., in regression it can be linear.</p>
  <p>Having multiple non-linear functions in your neural-net allows it to model (learn) more complex functions, so exploiting the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal function approximation</a> thing: indeed, assuming a correct architecture, loss function, and optimization procedure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T14:20:24.997Z
Answers.LastActivityDate: 2023-03-19T14:20:24.997Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not?
  The non-linear activation is required to scale the output.</p>
  
Tags: <neural-networks><backpropagation><unsupervised-learning><artificial-neuron>
Title: Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
UserDisplayName: Prabal Devkota
ThreadText: # Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
  
  (Asked by: username_0 on 2023-03-19)
  
  I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not?
  The non-linear activation is required to scale the output.
  
  **Tags:** neural-networks, backpropagation, unsupervised-learning, artificial-neuron
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-03-25. Score: 1
  
  A non-linear layer is a matrix multiplication followed by an activation function. A linear layer is just a matrix multiplication.
  
  You have a matrix multiplication, then another matrix multiplication, then an activation function.
  
  Two matrix multiplications in series are equivalent to one matrix multiplication, so your network is equivalent to one non-linear layer.
  
  
  ---
  
  ### Answer by username_2 on 2023-03-19. Score: 0
  
  You are speaking about generalized linear models (GLM)
  
  > **Comments:**
  > **username_0**: Do the model GLM follow semi linearity? I am new to this. Could you please help to provide detail on how it works?  
  
  ---
  
  ### Answer by username_3 on 2023-03-19. Score: 0
  
  If you have a `sigmoid` activation in your output layer, and linearities before, you can interpret it as a [logistic regression](https://en.wikipedia.org/wiki/Logistic_regression) model for binary classification (if output dimension is one.)
  
  Also, in practice, having multiple *linear* hidden layers is a waste of resources because you can define an equivalent linear layer whose weight matrix is the linear combination of those.
  
  Moreover, intermediate linear layers can be useful to "adjust" (i.e. expand or decrease) the output dimension of the previous layer (including the input layer.)
  
  In general, you want to have multiple non-linear layers (i.e. whatever layer with non-linear activation on top) in the middle of your model (i.e. as hidden layers) because the activation of the output layer is often (if not always) problem dependent: e.g., in regression it can be linear.
  
  Having multiple non-linear functions in your neural-net allows it to model (learn) more complex functions, so exploiting the [universal function approximation](https://en.wikipedia.org/wiki/Universal_approximation_theorem) thing: indeed, assuming a correct architecture, loss function, and optimization procedure.
  
  
  ---
  
  