Answers.Body: <blockquote>
  <p>Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?</p>
  </blockquote>
  <p>The main reason why in many cases, a CNN will outperform a fully-connected (FC) neural network, i.e. MLP, is grounded in symmetry. For example, a CNN responds very naturally to image translations. This behavior is called translational equivariance which arises from shifting the same weights (i.e. the convolutional kernel) over the space of an image. The result is a model that - when trained on images where an object is in the center - will generalize to images where an object is off-center. This happens even though such images are not in your training set. An MLP cannot do this, it would have be trained on a dataset containing the objects in all possible positions in order to generalize nicely.
  As a side note: Some architectures like the transformer use FC layers in the style of a convolution as well, because the exact same weights are applied to all input tokens individually. So a model doesn't have to be your typical 2D-ConvNet to be a type of CNN.</p>
  <blockquote>
  <p>FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.</p>
  </blockquote>
  <p>The model does not 'know' that it is overfitting because all it ever sees in training is the training dataset and an overfitted model performs great on this data. Another way to look at this is to view backpropagation as finding the path of least resistance to the optimal model. In many cases (especially with large models for small datasets) it is easier for the model to memorize the training samples instead of finding a very general solution.</p>
  <blockquote>
  <p>We know that fully connected means better.</p>
  </blockquote>
  <p>All layers have benefits and drawbacks, you can't really call any layer generally better, a fully-connected layer can do some things better than a CNN and other things (e.g. images) not so much.</p>
  <blockquote>
  <p>And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.</p>
  </blockquote>
  <p>An overfitted model might look nice in training, but as soon as you show the model novel samples, these are much more likely going to be misclassified. Especially messy real-world data will likely result in bad model performance when you actually want to use such a model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-27T19:35:33.247Z
Answers.Text: An MLP can learn just the same functions that CNN does. After all the CNN filters are just neurons aranged together. The big difference is that the weights are shared between the neurons. So you can just learn the same function with an MLP, but the neurons somehow (part of them) must learn the same weights.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-29T09:08:20.620Z
Answers.Text: Exactly, this is where the symmetry idea manifests: If you do not want to include all possible positions of objects in images in your trainingset, you choose a local operator that naturally models the underlying symmetry. In this case translational symmetry.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-17T00:43:35.677Z
Answers.LastActivityDate: 2023-02-17T00:43:35.677Z
Answers.UserDisplayName: Chillston
Body: <p>Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?</p>
  <p>FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.</p>
  <p>If eventually CNN will be flattened into FCNN, that's mean 2D layers (Conv2D, MaxPool, etc) of CNN is basically just 1D layers vector.</p>
  <p>Also CNN is not fully connected when doing convolution, except the kernel size is 1×1 with stride 1 which basically same as fully connected layer. We know that fully connected means better.</p>
  <p>And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-17T01:52:26.053Z
Comments.Text: Two posts of mine on Cross Validated seem related: https://stats.stackexchange.com/questions/271198/convolution-operator-in-cnn-and-how-it-differs-from-feed-forward-nn-operation/409172#409172 https://stats.stackexchange.com/questions/559113/milder-than-a-convolutional-neural-network-not-forcing-connections-to-be-perf $//$ Did you mean to included the [tag:fully-convolutional-networks] (FCN) tag? FCN and FCNN are not the same, and I think you mean FCNN.
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-17T10:40:55.473Z
Comments.Text: Do you know about the "no free lunch theorem"? Considering it, I think the question makes no sense, there is no better between layers, it all depends on the data and task.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-17T23:12:07.147Z
Comments.Text: @Dr.Snoopy assume using same data and same task.
Comments.UserDisplayName: Muhammad Ikhwan Perwira
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-17T23:27:57.503Z
Comments.Text: That assumption makes no sense, as these layers do not do the same task or receive the same input data.
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T03:50:28.977Z
Comments.Text: @Dr.Snoopy it's just preprocessing data issue, you can transform 1D data into 2D (image) so it compatible with CNN input and vice versa for FCNN
Comments.UserDisplayName: Muhammad Ikhwan Perwira
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T03:51:33.027Z
Comments.Text: @Dave My bad, I should have create new tag.
Comments.UserDisplayName: Muhammad Ikhwan Perwira
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T09:52:10.987Z
Comments.Text: No, that works only in one direction, there is no way to transform tabular data into images, there is no spatial relationship. Now I see why you ask this question, you have many misconceptions about basic ML.
Comments.UserDisplayName: Dr. Snoopy
Tags: <neural-networks><convolutional-neural-networks><fully-convolutional-networks>
Title: Why FCNN is not always better than CNN?
UserDisplayName: Muhammad Ikhwan Perwira
ThreadText: # Why FCNN is not always better than CNN?
  
  (Asked by: username_0 on 2023-02-16)
  
  Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?
  
  FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.
  
  If eventually CNN will be flattened into FCNN, that's mean 2D layers (Conv2D, MaxPool, etc) of CNN is basically just 1D layers vector.
  
  Also CNN is not fully connected when doing convolution, except the kernel size is 1×1 with stride 1 which basically same as fully connected layer. We know that fully connected means better.
  
  And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.
  
  **Tags:** neural-networks, convolutional-neural-networks, fully-convolutional-networks
  
  > **Comments:**
  > **username_1**: Two posts of mine on Cross Validated seem related: https://stats.stackexchange.com/questions/271198/convolution-operator-in-cnn-and-how-it-differs-from-feed-forward-nn-operation/409172#409172 https://stats.stackexchange.com/questions/559113/milder-than-a-convolutional-neural-network-not-forcing-connections-to-be-perf $//$ Did you mean to included the [tag:fully-convolutional-networks] (FCN) tag? FCN and FCNN are not the same, and I think you mean FCNN.  
  > **username_2**: Do you know about the "no free lunch theorem"? Considering it, I think the question makes no sense, there is no better between layers, it all depends on the data and task.  
  > **username_0**: @username_2 assume using same data and same task.  
  > **username_2**: That assumption makes no sense, as these layers do not do the same task or receive the same input data.  
  > **username_0**: @username_2 it's just preprocessing data issue, you can transform 1D data into 2D (image) so it compatible with CNN input and vice versa for FCNN  
  > **username_0**: @username_1 My bad, I should have create new tag.  
  > **username_2**: No, that works only in one direction, there is no way to transform tabular data into images, there is no spatial relationship. Now I see why you ask this question, you have many misconceptions about basic ML.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-02-17. Score: 2 | Accepted answer
  
  > Why Fully-Connected Neural Network is not always better than Convolutional Neural Network?
  
  The main reason why in many cases, a CNN will outperform a fully-connected (FC) neural network, i.e. MLP, is grounded in symmetry. For example, a CNN responds very naturally to image translations. This behavior is called translational equivariance which arises from shifting the same weights (i.e. the convolutional kernel) over the space of an image. The result is a model that - when trained on images where an object is in the center - will generalize to images where an object is off-center. This happens even though such images are not in your training set. An MLP cannot do this, it would have be trained on a dataset containing the objects in all possible positions in order to generalize nicely.
  As a side note: Some architectures like the transformer use FC layers in the style of a convolution as well, because the exact same weights are applied to all input tokens individually. So a model doesn't have to be your typical 2D-ConvNet to be a type of CNN.
  
  > FCNN is easily overfitting due to many params, then why didn't it reduce the params to reduce overfitting.
  
  The model does not 'know' that it is overfitting because all it ever sees in training is the training dataset and an overfitted model performs great on this data. Another way to look at this is to view backpropagation as finding the path of least resistance to the optimal model. In many cases (especially with large models for small datasets) it is easier for the model to memorize the training samples instead of finding a very general solution.
  
  > We know that fully connected means better.
  
  All layers have benefits and drawbacks, you can't really call any layer generally better, a fully-connected layer can do some things better than a CNN and other things (e.g. images) not so much.
  
  > And I think kernel size 1×1 and stride 1 in CNN (which basically equal same as FCNN) is performing better even though overfitting.
  
  An overfitted model might look nice in training, but as soon as you show the model novel samples, these are much more likely going to be misclassified. Especially messy real-world data will likely result in bad model performance when you actually want to use such a model.
  
  > **Comments:**
  > **username_4**: An MLP can learn just the same functions that CNN does. After all the CNN filters are just neurons aranged together. The big difference is that the weights are shared between the neurons. So you can just learn the same function with an MLP, but the neurons somehow (part of them) must learn the same weights.  
  > **username_3**: Exactly, this is where the symmetry idea manifests: If you do not want to include all possible positions of objects in images in your trainingset, you choose a local operator that naturally models the underlying symmetry. In this case translational symmetry.  
  
  ---
  
  