Answers.Body: <p>In general, when people do not explicitly state it, a vector <span class="math-container">$v \in \mathbb{R}^n$</span> is usually considered a "<a href="https://en.wikipedia.org/wiki/Row_and_column_vectors" rel="nofollow noreferrer">column vector</a>", that is, you can think of it as the matrix <span class="math-container">$v \in \mathbb{R}^{n \times 1}$</span> (that is, a matrix with <span class="math-container">$n$</span> rows and <span class="math-container">$1$</span> column). 
  If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "<a href="https://en.wikipedia.org/wiki/Row_and_column_vectors" rel="nofollow noreferrer">row vector</a>" (because it might be the case that the author is implicitly considering the vectors as row vectors).</p>
  
  <p>Having said that, you can multiply a vector <span class="math-container">$v \in \mathbb{R}^n$</span> by a matrix <span class="math-container">$A \in \mathbb{R}^{n \times m}$</span> <em>from the left</em>, that is, you can do <span class="math-container">$v^T A$</span>. Here, I considered <span class="math-container">$v$</span> has a column vector (that is, <span class="math-container">$v \in \mathbb{R}^{n \times 1}$</span>), even though I have not explicitly stated it: <em>you can and you often need to deduce this from the context</em>! If I transpose <span class="math-container">$v$</span>, I obtain <span class="math-container">$v^T \in \mathbb{R}^{1 \times n}$</span>, and, thus, you can see that you can indeed perform the operation <span class="math-container">$v^T A = u \in \mathbb{R}^{1 \times m}$</span>. Note that, at this point, <span class="math-container">$u$</span>, the vector resulting from the operation <span class="math-container">$v^T A$</span>, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply <span class="math-container">$v$</span> <em>from the right</em> of <span class="math-container">$A$</span>, because there is no way of making the dimensions match. Have a look at <a href="https://math.stackexchange.com/q/1615793/168764">this question</a>, if you do not know how to multiply a vector by a matrix from the left.</p>
  
  <p>Similarly, you can multiply <span class="math-container">$v \in \mathbb{R}^n$</span> by the matrix <span class="math-container">$B \in \mathbb{R}^{m \times n}$</span> only from the right. If <span class="math-container">$v$</span> is a column vector (that is, <span class="math-container">$v \in \mathbb{R}^{n \times 1}$</span>), you need to do <span class="math-container">$B v \in \mathbb{R}^{m \times 1}$</span>, but, if <span class="math-container">$v$</span> is a row vector (that is, <span class="math-container">$v \in \mathbb{R}^{1 \times n}$</span>), you will first need to transpose it, so that you can perform the operation: <span class="math-container">$B v^T\in \mathbb{R}^{m \times 1}$</span>.</p>
  
  <p>Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.</p>
  
  <p>To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T13:35:17.570Z
Answers.Text: @DuttaA I read the question. I tried to give an answer which is as general and thus, hopefully, useful as possible. In my answer, I explain all the details that the OP is confused about.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T13:40:39.600Z
Answers.Text: @DuttaA I will wait for the feedback of the OP. If he's not happy with my answer, I will modify it. If he is also confused about how to structure his weight matrix, then this is another question, which either can be asked separately, or maybe I could also answer it here.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T13:43:01.753Z
Answers.Text: @DuttaA Also note that the OP stated "The concept I’m have hard time understanding is multiplying vectors by a matrix". In my answer, I give him the general guidelines/rules to follow. I also address his statement " Im am confused because I assumed multiplying vector by matrix needs the same number of columns as well but have read that the matrixes need rows equal to the vectors columns.", so the confusion lies in how to multiply vectors by matrices: this is what I supposed from his/her statements.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T13:55:04.263Z
Answers.Text: @DuttaA To me, it was clear (apart maybe from the last paragraph). He's confused about how to multiply matrices with vectors.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T16:06:26.280Z
Answers.Text: Thank you nBro. I feel slightly more knowledgeable but don’t understand how weighted matrix don’t need equal columns. In a book example; The weighted matrix has 3 rows that represent each type of weight. The columns provide the weighted percentage for the  corresponding input(the 1x3 vector). If my matrix did not have a third column then the input vector would not have a percentage weighed for that third column input from vector. Also the the rows of my matrix represent different types of weights.If I only wanted to use two weights, Now I would have two row matrix. This means I cant multiply
Answers.UserDisplayName: sirshakir
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T16:25:14.540Z
Answers.Text: @zEyeland In a NN, weights are usually associated with connections between neurons. In a fully-connected feed-forward neural network, every neuron is connected to every other neuron. So, say, that you 3 neurons in the first layer and 3 neurons in the second layer, then you would need a $3 \times 3$ matrix, say $M$, to hold all $9$  "weights" of all possible connections between these neurons.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T16:25:17.973Z
Answers.Text: What the columns or rows represent in that matrix $M$ depends on the context: it might be that each row contains the weights associated with one neuron from the connections to all other neurons in the previous (or next) layer, or it might be that each column represents and holds such weights. It depends on the context (i.e. it may be explicitly stated or you need to deduce it from the math).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T16:30:36.803Z
Answers.Text: @zEyeland Please, if you have another question other than how matrix-vector multiplication works, then you should ask it separately and clearly.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-09T12:13:37.543Z
Answers.LastActivityDate: 2019-02-09T14:23:42.413Z
Answers.LastEditDate: 2019-02-09T14:23:42.413Z
Answers.UserDisplayName: nbro
Body: <p>I've recently started reading a book about deep learning. The book is titled "Grokking Deep Learning" (by Andrew W Trask). In chapter 3 (pages 44 and 45), it talks about multiplying vectors using dot product and element-wise multiplication. For instance, taking 3 scalar inputs (vector) and 3 vector weights (matrix) and multiplying. </p>
  
  <p>From my understanding, when multiplying vectors the size needs to be identical. The concept I have a hard time understanding is multiplying vectors by a matrix. The book gives an example of an 1x4 vector being multiplied by 4x3 matrix. The output is an 1x3 vector. I'm am confused because I assumed multiplying vector by matrix needs the same number of columns, but I have read that the matrices need rows equal to the vectors columns. </p>
  
  <p>If I do not have an equal number of columns, how does my deep learning algorithm multiply each input in my vector by a corresponding weight?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-09T12:06:03.713Z
Comments.Text: I the deep learning algorithm; I might have vector of three(columns) inputs. I also would have three weights. Each weight would have a vector of three columns to correspond With each input. These weights would be stored in matrix. The box gives an example of element wise multiplication with a 1x4 times 2x5 and states they don’t have same number of columns so it throws error. On the next page it give example of 1x4 times 4x3 and states the output is a 1x3 vector. This is contradicting.
Comments.UserDisplayName: sirshakir
Tags: <deep-learning><math>
Title: How are vectors and matrices multiplied in supervised machine learning?
UserDisplayName: sirshakir
ThreadText: # How are vectors and matrices multiplied in supervised machine learning?
  
  (Asked by: username_0 on 2019-02-09)
  
  I've recently started reading a book about deep learning. The book is titled "Grokking Deep Learning" (by Andrew W Trask). In chapter 3 (pages 44 and 45), it talks about multiplying vectors using dot product and element-wise multiplication. For instance, taking 3 scalar inputs (vector) and 3 vector weights (matrix) and multiplying.
  
  From my understanding, when multiplying vectors the size needs to be identical. The concept I have a hard time understanding is multiplying vectors by a matrix. The book gives an example of an 1x4 vector being multiplied by 4x3 matrix. The output is an 1x3 vector. I'm am confused because I assumed multiplying vector by matrix needs the same number of columns, but I have read that the matrices need rows equal to the vectors columns.
  
  If I do not have an equal number of columns, how does my deep learning algorithm multiply each input in my vector by a corresponding weight?
  
  **Tags:** deep-learning, math
  > **Comments:**
  > **username_0**: I the deep learning algorithm; I might have vector of three(columns) inputs. I also would have three weights. Each weight would have a vector of three columns to correspond With each input. These weights would be stored in matrix. The box gives an example of element wise multiplication with a 1x4 times 2x5 and states they don’t have same number of columns so it throws error. On the next page it give example of 1x4 times 4x3 and states the output is a 1x3 vector. This is contradicting.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-09. Score: 1
  
  In general, when people do not explicitly state it, a vector $v \in \mathbb{R}^n$ is usually considered a "[column vector](https://en.wikipedia.org/wiki/Row_and_column_vectors)", that is, you can think of it as the matrix $v \in \mathbb{R}^{n \times 1}$ (that is, a matrix with $n$ rows and $1$ column).
  If it is not explicitly stated and you assume that the given vector is a column vector, but the dimensions do not match, then you should check that the dimensions match if you consider the given vector as a "[row vector](https://en.wikipedia.org/wiki/Row_and_column_vectors)" (because it might be the case that the author is implicitly considering the vectors as row vectors).
  
  Having said that, you can multiply a vector $v \in \mathbb{R}^n$ by a matrix $A \in \mathbb{R}^{n \times m}$ *from the left*, that is, you can do $v^T A$. Here, I considered $v$ has a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), even though I have not explicitly stated it: *you can and you often need to deduce this from the context*! If I transpose $v$, I obtain $v^T \in \mathbb{R}^{1 \times n}$, and, thus, you can see that you can indeed perform the operation $v^T A = u \in \mathbb{R}^{1 \times m}$. Note that, at this point, $u$, the vector resulting from the operation $v^T A$, is actually considered a matrix, but you can still use it as a vector, if you need and that is permitted according to the mathematical rules of the operations you need to perform. Note also that I cannot multiply $v$ *from the right* of $A$, because there is no way of making the dimensions match. Have a look at [this question](https://math.stackexchange.com/q/1615793/168764), if you do not know how to multiply a vector by a matrix from the left.
  
  Similarly, you can multiply $v \in \mathbb{R}^n$ by the matrix $B \in \mathbb{R}^{m \times n}$ only from the right. If $v$ is a column vector (that is, $v \in \mathbb{R}^{n \times 1}$), you need to do $B v \in \mathbb{R}^{m \times 1}$, but, if $v$ is a row vector (that is, $v \in \mathbb{R}^{1 \times n}$), you will first need to transpose it, so that you can perform the operation: $B v^T\in \mathbb{R}^{m \times 1}$.
  
  Furthermore, note that, if you multiply a vector by a matrix from the left, that vector needs to be a "row vector", so, if you initially assume that the vector is a column vector (or that is explicitly stated), you will need to transpose it first, before the multiplication. However, if the vector is already a row vector, you won't have to transpose it. Similarly, if you multiply a vector by a matrix from the right, you will need a column vector.
  
  To conclude, you can multiply a vector either from the left or right of a matrix, but you need to make sure that the dimensions match: if you multiply from the left, you will need to check that the dimensions of the vector match the number of the rows of the matrix; if you multiply the vector from the right of the matrix, you will need to check that the dimensions of the vector match the number of columns of the matrix.
  
  > **Comments:**
  > **username_1**: @DuttaA I read the question. I tried to give an answer which is as general and thus, hopefully, useful as possible. In my answer, I explain all the details that the OP is confused about.  
  > **username_1**: @DuttaA I will wait for the feedback of the OP. If he's not happy with my answer, I will modify it. If he is also confused about how to structure his weight matrix, then this is another question, which either can be asked separately, or maybe I could also answer it here.  
  > **username_1**: @DuttaA Also note that the OP stated "The concept I’m have hard time understanding is multiplying vectors by a matrix". In my answer, I give him the general guidelines/rules to follow. I also address his statement " Im am confused because I assumed multiplying vector by matrix needs the same number of columns as well but have read that the matrixes need rows equal to the vectors columns.", so the confusion lies in how to multiply vectors by matrices: this is what I supposed from his/her statements.  
  > **username_1**: @DuttaA To me, it was clear (apart maybe from the last paragraph). He's confused about how to multiply matrices with vectors.  
  > **username_0**: Thank you nBro. I feel slightly more knowledgeable but don’t understand how weighted matrix don’t need equal columns. In a book example; The weighted matrix has 3 rows that represent each type of weight. The columns provide the weighted percentage for the  corresponding input(the 1x3 vector). If my matrix did not have a third column then the input vector would not have a percentage weighed for that third column input from vector. Also the the rows of my matrix represent different types of weights.If I only wanted to use two weights, Now I would have two row matrix. This means I cant multiply  
  > **username_1**: @zEyeland In a NN, weights are usually associated with connections between neurons. In a fully-connected feed-forward neural network, every neuron is connected to every other neuron. So, say, that you 3 neurons in the first layer and 3 neurons in the second layer, then you would need a $3 \times 3$ matrix, say $M$, to hold all $9$  "weights" of all possible connections between these neurons.  
  > **username_1**: What the columns or rows represent in that matrix $M$ depends on the context: it might be that each row contains the weights associated with one neuron from the connections to all other neurons in the previous (or next) layer, or it might be that each column represents and holds such weights. It depends on the context (i.e. it may be explicitly stated or you need to deduce it from the math).  
  > **username_1**: @zEyeland Please, if you have another question other than how matrix-vector multiplication works, then you should ask it separately and clearly.  
  
  ---
  
  