Answers.Body: <p>Yes, typically, a fully connected layer is an affine transformation, which can or not be followed by a non-linear activation function, but, in many (if not most) cases, it's followed by a non-linearity, such as ReLU, sigmoid, or tanh (<a href="https://ai.stackexchange.com/a/31722/2444">an exception is when you do regression</a>), which is what makes the neural network be able to <a href="https://ai.stackexchange.com/a/13319/2444">approximate non-linear/complicated functions</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T00:20:23.567Z
Answers.LastActivityDate: 2021-09-20T00:20:23.567Z
Answers.UserDisplayName: nbro
Body: <p>We generally encounter the following statement several times</p>
  <blockquote>
  <p>The input vector is first fed into a <strong>fully connected layer</strong>......</p>
  </blockquote>
  <p>Since linear activation functions, such as identity function, can so considered as an activation functions, a fully connected layer can be considered just as an Affine transformation if the fully connected layer uses linear activation function.</p>
  <p>So, in theory, a fully connected layer can refer to the following</p>
  <ol>
  <li>Just an affine transformation</li>
  <li>Affine transformation followed by a nonlinear activation function</li>
  </ol>
  <p>Do authors generally choose to use &quot;fully connected layer&quot; for case 2 only or for both cases 1 and 2?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-16T08:06:29.373Z
Comments.Text: Affine transformations have this property that multiple affine transformations are equivalent to one affine transformation, so you gain nothing by stacking such layers.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-16T08:23:59.830Z
Comments.Text: @user253751 consider a single one.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-16T20:22:44.087Z
Comments.Text: According to my experience, it refers to the 2nd case. When you're talking of neural networks, it's implicitly assumed you use a non-linear activation function at the end of each layer.
Comments.UserDisplayName: SpiderRico
Tags: <neural-networks><terminology>
Title: Do authors generally use fully connected layer instead of affine transformation?
UserDisplayName: hanugm
ThreadText: # Do authors generally use fully connected layer instead of affine transformation?
  
  (Asked by: username_0 on 2021-09-16)
  
  We generally encounter the following statement several times
  
  > The input vector is first fed into a **fully connected layer**......
  
  Since linear activation functions, such as identity function, can so considered as an activation functions, a fully connected layer can be considered just as an Affine transformation if the fully connected layer uses linear activation function.
  
  So, in theory, a fully connected layer can refer to the following
  
  1. Just an affine transformation
  2. Affine transformation followed by a nonlinear activation function
  
  Do authors generally choose to use "fully connected layer" for case 2 only or for both cases 1 and 2?
  
  **Tags:** neural-networks, terminology
  > **Comments:**
  > **username_1**: Affine transformations have this property that multiple affine transformations are equivalent to one affine transformation, so you gain nothing by stacking such layers.  
  > **username_0**: @user253751 consider a single one.  
  > **username_2**: According to my experience, it refers to the 2nd case. When you're talking of neural networks, it's implicitly assumed you use a non-linear activation function at the end of each layer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-20. Score: 1 | Accepted answer
  
  Yes, typically, a fully connected layer is an affine transformation, which can or not be followed by a non-linear activation function, but, in many (if not most) cases, it's followed by a non-linearity, such as ReLU, sigmoid, or tanh ([an exception is when you do regression](https://ai.stackexchange.com/a/31722/2444)), which is what makes the neural network be able to [approximate non-linear/complicated functions](https://ai.stackexchange.com/a/13319/2444).
  
  
  ---
  
  