Answers.Body: <p>I don't think there's a &quot;standard way&quot; of expressing the forward pass: you use the transpose when you need to use it, and this depends on how you define the weights and inputs matrices, and on the architecture of your neural network. For example, in a fully connected feedforward neural network, you know that every neuron in the previous layer is connected to every neuron in the current layer, so, as long as this is satisfied when you multiply the matrices, it does not matter whether you use transposes or not, and I don't think that, in computational terms, it makes any difference if you use transposes or not. (By the way, if you are writing something, I suggest that you always specify the dimensions of your matrices and your conventions).</p>
  <p>Of course, if you want to use a library like TensorFlow, you will probably need to follow the conventions of the library, but this is another story.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-10T03:54:05.653Z
Answers.LastActivityDate: 2020-08-10T03:54:05.653Z
Answers.UserDisplayName: nbro
Body: <p>I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard neural network as <code>np.dot(x, W)</code> and sometimes I see it as <code>np.dot(W.T, x)</code> and sometimes <code>np.dot(W, x)</code>.</p>
  <p>Take this image for example. They represent the input data as a matrix of <code>[NxD]</code> and weight data as <code>[DxH]</code> where H is the number of neurons in the hidden layer. This seems the most natural since input data will often be in tabular format with rows as samples and columns as features.</p>
  <p><a href="https://i.sstatic.net/wUNV7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wUNV7.png" alt="Example" /></a></p>
  <p>Now an example from the CS231n course notes. They talk about this below example and cite the code used to compute it as:</p>
  <pre><code>f = lambda x: 1.0/(1.0 + np.exp(-x)) # activation function (use sigmoid)
  x = np.random.randn(3, 1) # random input vector of three numbers (3x1)
  h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations (4x1)
  h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations (4x1)
  out = np.dot(W3, h2) + b3 # output neuron (1x1)
  </code></pre>
  <p>Where <code>W</code> is <code>[4x3]</code> and x is <code>[3x1]</code>. I would expect the weight matrix to have dimensions equal to [n_features, n_hidden_neurons] but in this example it just seems like they transposed it naturally before it was used.</p>
  <p><a href="https://i.sstatic.net/FHJ5p.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FHJ5p.jpg" alt="Example2" /></a></p>
  <p>I guess I am just confused about general nomenclature in how data should be shaped and used consistently when computing neural network forward passes. Sometimes I see transpose, sometimes I don't. Is there a standard, preferred way to represent data in accordance to a diagram like these This question may be silly but I just wanted to discuss it a bit. Thank you.</p>
  
Tags: <neural-networks><python><feedforward-neural-networks><forward-pass>
Title: What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?
UserDisplayName: Coldchain9
ThreadText: # What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?
  
  (Asked by: username_0 on 2020-08-09)
  
  I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard neural network as np.dot(x, W) and sometimes I see it as np.dot(W.T, x) and sometimes np.dot(W, x).
  Take this image for example. They represent the input data as a matrix of [NxD] and weight data as [DxH] where H is the number of neurons in the hidden layer. This seems the most natural since input data will often be in tabular format with rows as samples and columns as features.
  
  Now an example from the CS231n course notes. They talk about this below example and cite the code used to compute it as:
  f = lambda x: 1.0/(1.0 + np.exp(-x)) # activation function (use sigmoid)
  x = np.random.randn(3, 1) # random input vector of three numbers (3x1)
  h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations (4x1)
  h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations (4x1)
  out = np.dot(W3, h2) + b3 # output neuron (1x1)
  
  Where W is [4x3] and x is [3x1]. I would expect the weight matrix to have dimensions equal to [n_features, n_hidden_neurons] but in this example it just seems like they transposed it naturally before it was used.
  
  I guess I am just confused about general nomenclature in how data should be shaped and used consistently when computing neural network forward passes. Sometimes I see transpose, sometimes I don't. Is there a standard, preferred way to represent data in accordance to a diagram like these This question may be silly but I just wanted to discuss it a bit. Thank you.
  
  **Tags:** neural-networks, python, feedforward-neural-networks, forward-pass
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-10. Score: 1
  
  I don't think there's a "standard way" of expressing the forward pass: you use the transpose when you need to use it, and this depends on how you define the weights and inputs matrices, and on the architecture of your neural network. For example, in a fully connected feedforward neural network, you know that every neuron in the previous layer is connected to every neuron in the current layer, so, as long as this is satisfied when you multiply the matrices, it does not matter whether you use transposes or not, and I don't think that, in computational terms, it makes any difference if you use transposes or not. (By the way, if you are writing something, I suggest that you always specify the dimensions of your matrices and your conventions).
  
  Of course, if you want to use a library like TensorFlow, you will probably need to follow the conventions of the library, but this is another story.
  
  
  ---
  
  