Answers.Body: <p>Your intuition of possibly different positioning of a same image causing different ordering of each channel's pixel value to feed into a artificial neural network (ANN) makes sense, but the <em>relative</em> order between local neighboring pixels still keeps intact thus it's usually not a fatal problem. So long as most local input information is kept intact, as a universal function approximator ANNs can adjust and keep their relative result intact too. Furthermore the specialized highly performant <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network" rel="nofollow noreferrer">convolutional neural network</a> (CNN) was designed to take advantage of your mentioned the conspicuous space invariance property of images.</p>
  <blockquote>
  <p>CNNs are also known as shift invariant or space invariant artificial neural networks (SIANN), based on the shared-weight architecture of the convolution kernels or filters that slide along input features and provide translation-equivariant responses known as feature maps... Because the degree of model overfitting is determined by both its power and the amount of training it receives, providing a convolutional network with more training examples can reduce overfitting... Because there is often not enough available data to train...  two approaches are to either generate new data from scratch (if possible) or perturb existing data to create new ones...  For example, input images can be cropped, rotated, or rescaled to create new examples with the same labels as the original training set.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-02T08:36:07.017Z
Answers.LastActivityDate: 2024-08-02T08:36:07.017Z
Answers.UserDisplayName: cinch
Body: <p>I've learned some machine learning networks, and I think they're not that interesting. Sometimes I feel like they're just messing around with formulas and creating something that doesn't make sense. And some tutorials only tell you how to do it, but not why it's designed that way.</p>
  <p>This is especially true in neural networks and AI. It's frustrating. I've been trying to understand the relationship between fully connected neural networks and matrix multiplication, but I've been getting some confusing feelings that I couldn't quite put into words. But now I think I've found some problems, so I'd like to ask and see how you respond.</p>
  <p>Below are some simple examples of matrix multiplication, which are similar to the so-called &quot;neural networks&quot; that simulate the human brain. There are some circles, and then the input layer.</p>
  <p>The key point is the input layer. I think there's a problem here.</p>
  <p>For example, let's say we have a 3x5 matrix X and a 5x10 matrix W. We can ignore the bias term b for now.
  <a href="https://i.sstatic.net/bZNApBBU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bZNApBBU.png" alt="enter image description here" /></a>
  W is the weight matrix that transforms the input X into the neural network. Let's look at how matrix multiplication simulates the neural network.
  <a href="https://i.sstatic.net/bZBFWzVU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bZBFWzVU.png" alt="enter image description here" /></a>
  For example, the first row of X is X00, X01,..., X04. According to matrix multiplication, this row needs to be multiplied element-wise with each column of W, and then summed to get a new row, which is Y00, Y01,..., Y09.</p>
  <p>Let's look at the neural network corresponding to W. This layer has 5 neurons, because each row represents a neuron, and each neuron has 10 inputs, because there are 10 columns.</p>
  <p>So when we multiply the first row of X with each column of W, we find that X00 is always multiplied with W00, W01,..., W09, which means that X00 is always input to the first neuron's all inputs.</p>
  <p>Do you think this is correct?</p>
  <p>Assuming X is an image, then X00 is just one pixel of the image. This pixel is always input to the first neuron's all inputs, but never to the second neuron, for example, W10, because one row is just one neuron.</p>
  <p>So my question is, if X is an image, don't you think that X00 is important for the second, third, fourth neurons, and so on? Because of matrix multiplication, X00 can't be input to any other neurons in the first layer.</p>
  <p>I see this and I think there's a problem.</p>
  <p>If we manually rotate the image X by 180 degrees, then X00 will be input to W40, which is a different neuron.</p>
  <p>If the network produces a very different output, then there's a problem, because rotating an image by 180 degrees shouldn't change its content.</p>
  <p>So?</p>
  <p>I've looked at the source code of LLaMA3 and other neural networks, and every time I see this, I get confused. Since there's uncertainty, why not just combine all possible situations? I think that's the only way to make sure that the network is correct. Instead of programmers writing code that they think should work, but only considering a limited perspective, and then training the AI.</p>
  <p>You can't guarantee that your own code is correct, you can only try to include all possible situations and then rely on computational power, luck, and other factors.</p>
  <p>So, regarding the above situation, my idea is that X00 should be input to all neurons' all inputs, not just some of them.</p>
  <p>Am I correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-02T08:14:30.407Z
Comments.Text: Can you put your **specific question** in the title? If you have multiple questions, create multiple posts
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks>
Title: About the Matrix Multiplication in Fully Connected Neural Networks
UserDisplayName: shang
ThreadText: # About the Matrix Multiplication in Fully Connected Neural Networks
  
  (Asked by: username_0 on 2024-08-01)
  
  I've learned some machine learning networks, and I think they're not that interesting. Sometimes I feel like they're just messing around with formulas and creating something that doesn't make sense. And some tutorials only tell you how to do it, but not why it's designed that way.
  
  This is especially true in neural networks and AI. It's frustrating. I've been trying to understand the relationship between fully connected neural networks and matrix multiplication, but I've been getting some confusing feelings that I couldn't quite put into words. But now I think I've found some problems, so I'd like to ask and see how you respond.
  
  Below are some simple examples of matrix multiplication, which are similar to the so-called "neural networks" that simulate the human brain. There are some circles, and then the input layer.
  
  The key point is the input layer. I think there's a problem here.
  
  For example, let's say we have a 3x5 matrix X and a 5x10 matrix W. We can ignore the bias term b for now.
  [![enter image description here](https://i.sstatic.net/bZNApBBU.png)](https://i.sstatic.net/bZNApBBU.png)
  W is the weight matrix that transforms the input X into the neural network. Let's look at how matrix multiplication simulates the neural network.
  [![enter image description here](https://i.sstatic.net/bZBFWzVU.png)](https://i.sstatic.net/bZBFWzVU.png)
  For example, the first row of X is X00, X01,..., X04. According to matrix multiplication, this row needs to be multiplied element-wise with each column of W, and then summed to get a new row, which is Y00, Y01,..., Y09.
  
  Let's look at the neural network corresponding to W. This layer has 5 neurons, because each row represents a neuron, and each neuron has 10 inputs, because there are 10 columns.
  
  So when we multiply the first row of X with each column of W, we find that X00 is always multiplied with W00, W01,..., W09, which means that X00 is always input to the first neuron's all inputs.
  
  Do you think this is correct?
  
  Assuming X is an image, then X00 is just one pixel of the image. This pixel is always input to the first neuron's all inputs, but never to the second neuron, for example, W10, because one row is just one neuron.
  
  So my question is, if X is an image, don't you think that X00 is important for the second, third, fourth neurons, and so on? Because of matrix multiplication, X00 can't be input to any other neurons in the first layer.
  
  I see this and I think there's a problem.
  
  If we manually rotate the image X by 180 degrees, then X00 will be input to W40, which is a different neuron.
  
  If the network produces a very different output, then there's a problem, because rotating an image by 180 degrees shouldn't change its content.
  
  So?
  
  I've looked at the source code of LLaMA3 and other neural networks, and every time I see this, I get confused. Since there's uncertainty, why not just combine all possible situations? I think that's the only way to make sure that the network is correct. Instead of programmers writing code that they think should work, but only considering a limited perspective, and then training the AI.
  
  You can't guarantee that your own code is correct, you can only try to include all possible situations and then rely on computational power, luck, and other factors.
  
  So, regarding the above situation, my idea is that X00 should be input to all neurons' all inputs, not just some of them.
  
  Am I correct?
  
  **Tags:** convolutional-neural-networks
  > **Comments:**
  > **username_1**: Can you put your **specific question** in the title? If you have multiple questions, create multiple posts  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-08-02. Score: 0
  
  Your intuition of possibly different positioning of a same image causing different ordering of each channel's pixel value to feed into a artificial neural network (ANN) makes sense, but the *relative* order between local neighboring pixels still keeps intact thus it's usually not a fatal problem. So long as most local input information is kept intact, as a universal function approximator ANNs can adjust and keep their relative result intact too. Furthermore the specialized highly performant [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) (CNN) was designed to take advantage of your mentioned the conspicuous space invariance property of images.
  
  > CNNs are also known as shift invariant or space invariant artificial neural networks (SIANN), based on the shared-weight architecture of the convolution kernels or filters that slide along input features and provide translation-equivariant responses known as feature maps... Because the degree of model overfitting is determined by both its power and the amount of training it receives, providing a convolutional network with more training examples can reduce overfitting... Because there is often not enough available data to train... two approaches are to either generate new data from scratch (if possible) or perturb existing data to create new ones... For example, input images can be cropped, rotated, or rescaled to create new examples with the same labels as the original training set.
  
  
  ---
  
  