Answers.Body: <p>This is a good example of what happens when you take text out of context. </p>
  
  <p>The passage that was added in the edited question makes a difference, but it's not quite sufficient, and it doesn't help that the notation is all over the place. I found the textbook and the relevant passages (Section 2.3, p. 10-11). Here is a quick attempt at an explanation.</p>
  
  <p>The authors call <em>X</em> a <strong>variable</strong> (either scalar or vector) with <strong>components</strong> <em>X<sub>j</sub></em>, but later in the paragraph they refer to <em>X<sub>j</sub></em> as a <strong>variable</strong> (which I think is the correct notation). Instead of "variable" with "components", think of <em>X</em> as a set of <em>p</em> variables (in the normal sense of a variable, such as temperature, the price of shares, etc.). You can put multiple variables <em>x<sub>i</sub></em> (<em>i = 1...p</em>) in a vector <em>X</em> and call each <em>instance</em> of <em>X</em> an <em>observation</em>. In other words, an observation is a set of measured values for <strong>all</strong> variables <em>x<sub>i</sub></em>.</p>
  
  <p>Assume you have made <em>N</em> observations. Arrange your observations in a matrix with <em>N</em> rows and <em>p</em> columns, where each row represents a single observation (an <em>instance</em> of <em>X</em>).</p>
  
  <p>Now also assume that you are trying to find the relation between your <strong>input</strong> variables <em>x<sub>i</sub></em> and a <em>different</em> set of variables <em>y<sub>k</sub></em>, called <em>dependent</em> or <em>response</em> variables. In general, the variables <em>x<sub>i</sub></em> and <em>y<sub>k</sub></em> are <strong>measured</strong>, and the model is simply trying to extract the relation between the input and dependent variables so you can then <em>predict</em> the latter from the former.</p>
  
  <p>As a side note, observations are usually denoted with <strong>superscripts</strong> (<em>x<sup>(n)</sup></em>) and variables with <strong>subscripts</strong> (<em>x<sub>i</sub></em>) so there is no confusion about which is which. <em>x<sub>i</sub><sup>(n)</sup></em> is the <em>n</em><sup>th</sup> observed (measured) value of variable <em>x<sub>i</sub></em>.</p>
  
  <p>In your case, you have a single dependent variable <em>y</em> and <em>p</em> input variables <em>x<sub>i</sub></em> (<em>i = 1...p</em>). Assuming that their relation is linear (note: in many cases this assumption is <em>not</em> justified), we can assign weights ("importance") to each variable and try to find out those weights from measurements. In your case, the weights are denoted with <em>beta<sub>i</sub></em> (so the "importance" of variable <em>x<sub>i</sub></em> is <em>beta<sub>i</sub></em>; note the same subscript).</p>
  
  <p>If you have <em>N</em> observations of your dependent variable <em>y</em>, then you can arrange them in a <strong>column</strong>, just like the observations for your input variables. Note that we still have a <em>single</em> dependent variable, so essentially <em>y</em> is a <strong>scalar</strong>, but the <em>N</em> <em>observations</em> of that scalar form an <em>N</em>-dimensional <em>column</em> vector.</p>
  
  <p>Now multiply the <em>N x p</em> input matrix by the <em>p</em>-dimensional <strong>column</strong> vector of weights <em>beta</em>. What you get is a <strong>column</strong> vector of <em>N</em> <strong>predicted values</strong> for <em>y</em> (<em>\hat{y}</em>). The difference of the <em>N</em>-dimensional column vector of predicted values <em>\hat{y}</em> and the <em>N</em>-dimensional column vector <em>y</em> of measured values is the <strong>error</strong> (which is minimised with the least squares method).</p>
  
  <p>If you have <em>q</em> dependent variables <em>y<sub>i</sub></em> (<em>i = 1...q</em>), each of them would generally have its own set of weights <em>beta</em>. In other words, the relation between the input variables and <em>each</em> <em>y<sub>i</sub></em> will be <em>different</em>. The dependent variables can also be arranged in a matrix (just like the input matrix), and its dimensionality will be <em>N x q</em>, where <em>q</em>. In that case, the <em>beta</em> matrix will <strong>not be a single <em>N</em>-dimensional column vector but an <em>N x q</em> matrix</strong>. Each column in the <em>beta</em> matrix gives the relation between the input variables and the <em>q</em><sup>th</sup> dependent variable <em>y<sub>q</sub></em>. In that respect, a single observation of all variables <em>y<sub>i</sub></em> will be a <em>row</em> vector in the <em>y</em> matrix.</p>
  
  <p>I hope that this clarifies things up a bit. In summary, the explanation in the textbook is correct, but the notation is hard to follow and at times plain misleading (as in the case of variable and component at the beginning of the section). Honestly, you can get a much more intuitive explanation of linear regression from the <a href="https://en.wikipedia.org/wiki/Linear_regression" rel="nofollow noreferrer">Wikipedia</a> article.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-03T14:12:32.950Z
Answers.LastActivityDate: 2018-07-03T14:12:32.950Z
Answers.UserDisplayName: cantordust
Body: <p>While reading about least squares implementation for machine learning I came across this passage in the following two photos:
  <a href="https://i.sstatic.net/rbHEv.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rbHEv.jpg" alt="photo1"></a>
  <a href="https://i.sstatic.net/FR8vr.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FR8vr.jpg" alt="photo 2"></a></p>
  
  <p>Perhaps I’m misinterpreting the meaning of beta  but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y}  would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted. </p>
  
  <p>Can someone provide clarification?</p>
  
  <p>Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:</p>
  
  <p><a href="https://i.sstatic.net/RGXdp.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RGXdp.jpg" alt="3rd"></a>
  <a href="https://i.sstatic.net/47qQn.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/47qQn.jpg" alt="4th"></a></p>
  
  <p>Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?</p>
  
  <p>Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, &amp; Friedman.  </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:10:06.160Z
Comments.Text: The text says `Here we are modeling a single output`, so `K = 1`. In that case, `\hat{Y}` would be a `1 x 1` vector. You could argue that this is not technically a scalar, but this is a minor point. If you are modelling more than one output, `K` would be greater than `1` and `\hat{Y}` would be a `1 x K` row vector where `K` is the number of outputs.
Comments.UserDisplayName: cantordust
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:13:32.600Z
Comments.Text: @cantordust thank you for your reply. Is this a typical convention, to have Y as a row vector? I only ask because I would expect it to be written as Y^T. But I’m just beginning my reading and haven’t seen applications enough to verify. In a linear algebra context, one would expect Y to be a column vector, but perhaps convention is different here?
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:20:45.177Z
Comments.Text: I suspect that the reason why they have omitted the `^T` is that it doesn't matter for a scalar (`a` happens to be the same as `a^T` when `a` is a `1 x 1` vector). But in principle you are right that it should be `Y^T`. This is kind of sloppy writing. They have also missed the `^` in the last sentence - that should be `\hat{\beta}`.
Comments.UserDisplayName: cantordust
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:24:22.843Z
Comments.Text: @cantordust I noticed that as well and was left wondering if that beta was different from \hat{\beta} but assumed it was the same. There is another section on the preceding page regarding their matrix notation which I found confusing from a linear algebra context. I’ll perhaps update the question to include it.
Comments.UserDisplayName: Hanzy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:26:36.020Z
Comments.Text: Sadly, modern textbooks are rushed out the door without being proofread properly...
Comments.UserDisplayName: cantordust
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-07-03T03:43:59.640Z
Comments.Text: @cantordust I’ve added a passage on the matrix notation in the book that I also find confusing. But perhaps it actually is consistent and I just misunderstand the meaning. Maybe those passages can provide more insight?
Comments.UserDisplayName: Hanzy
Tags: <machine-learning><linear-regression>
Title: Matrix Dimension for Linear regression coefficients
UserDisplayName: Hanzy
ThreadText: # Matrix Dimension for Linear regression coefficients
  
  (Asked by: username_0 on 2018-07-03)
  
  While reading about least squares implementation for machine learning I came across this passage in the following two photos:
  [![photo1](https://i.sstatic.net/rbHEv.jpg)](https://i.sstatic.net/rbHEv.jpg)
  [![photo 2](https://i.sstatic.net/FR8vr.jpg)](https://i.sstatic.net/FR8vr.jpg)
  
  Perhaps I’m misinterpreting the meaning of beta but if X^T has dimension 1 x p and beta has dimension p x K, then hat{Y} would have dimension 1 x K and would be a row vector. According to the text, vectors are assumed column vectors unless otherwise noted.
  
  Can someone provide clarification?
  
  Edit: the matrix notation in this text confuses me. The pages preceding the above passage read:
  
  [![3rd](https://i.sstatic.net/RGXdp.jpg)](https://i.sstatic.net/RGXdp.jpg)
  [![4th](https://i.sstatic.net/47qQn.jpg)](https://i.sstatic.net/47qQn.jpg)
  
  Should the matrix referenced not have dimensions p x N, assuming a p-vector is a column vector with p elements?
  
  Note: The passage is taken from “Elements of Statistical Learning” by Hastie, Tibshirani, & Friedman.
  
  **Tags:** machine-learning, linear-regression
  > **Comments:**
  > **username_1**: The text says `Here we are modeling a single output`, so `K = 1`. In that case, `\hat{Y}` would be a `1 x 1` vector. You could argue that this is not technically a scalar, but this is a minor point. If you are modelling more than one output, `K` would be greater than `1` and `\hat{Y}` would be a `1 x K` row vector where `K` is the number of outputs.  
  > **username_0**: @cantordust thank you for your reply. Is this a typical convention, to have Y as a row vector? I only ask because I would expect it to be written as Y^T. But I’m just beginning my reading and haven’t seen applications enough to verify. In a linear algebra context, one would expect Y to be a column vector, but perhaps convention is different here?  
  > **username_1**: I suspect that the reason why they have omitted the `^T` is that it doesn't matter for a scalar (`a` happens to be the same as `a^T` when `a` is a `1 x 1` vector). But in principle you are right that it should be `Y^T`. This is kind of sloppy writing. They have also missed the `^` in the last sentence - that should be `\hat{\beta}`.  
  > **username_0**: @cantordust I noticed that as well and was left wondering if that beta was different from \hat{\beta} but assumed it was the same. There is another section on the preceding page regarding their matrix notation which I found confusing from a linear algebra context. I’ll perhaps update the question to include it.  
  > **username_1**: Sadly, modern textbooks are rushed out the door without being proofread properly...  
  > **username_0**: @cantordust I’ve added a passage on the matrix notation in the book that I also find confusing. But perhaps it actually is consistent and I just misunderstand the meaning. Maybe those passages can provide more insight?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-03. Score: 1 | Accepted answer
  
  This is a good example of what happens when you take text out of context.
  
  The passage that was added in the edited question makes a difference, but it's not quite sufficient, and it doesn't help that the notation is all over the place. I found the textbook and the relevant passages (Section 2.3, p. 10-11). Here is a quick attempt at an explanation.
  
  The authors call *X* a **variable** (either scalar or vector) with **components** *Xj*, but later in the paragraph they refer to *Xj* as a **variable** (which I think is the correct notation). Instead of "variable" with "components", think of *X* as a set of *p* variables (in the normal sense of a variable, such as temperature, the price of shares, etc.). You can put multiple variables *xi* (*i = 1...p*) in a vector *X* and call each *instance* of *X* an *observation*. In other words, an observation is a set of measured values for **all** variables *xi*.
  
  Assume you have made *N* observations. Arrange your observations in a matrix with *N* rows and *p* columns, where each row represents a single observation (an *instance* of *X*).
  
  Now also assume that you are trying to find the relation between your **input** variables *xi* and a *different* set of variables *yk*, called *dependent* or *response* variables. In general, the variables *xi* and *yk* are **measured**, and the model is simply trying to extract the relation between the input and dependent variables so you can then *predict* the latter from the former.
  
  As a side note, observations are usually denoted with **superscripts** (*x(n)*) and variables with **subscripts** (*xi*) so there is no confusion about which is which. *xi(n)* is the *n*th observed (measured) value of variable *xi*.
  
  In your case, you have a single dependent variable *y* and *p* input variables *xi* (*i = 1...p*). Assuming that their relation is linear (note: in many cases this assumption is *not* justified), we can assign weights ("importance") to each variable and try to find out those weights from measurements. In your case, the weights are denoted with *betai* (so the "importance" of variable *xi* is *betai*; note the same subscript).
  
  If you have *N* observations of your dependent variable *y*, then you can arrange them in a **column**, just like the observations for your input variables. Note that we still have a *single* dependent variable, so essentially *y* is a **scalar**, but the *N* *observations* of that scalar form an *N*-dimensional *column* vector.
  
  Now multiply the *N x p* input matrix by the *p*-dimensional **column** vector of weights *beta*. What you get is a **column** vector of *N* **predicted values** for *y* (*\hat{y}*). The difference of the *N*-dimensional column vector of predicted values *\hat{y}* and the *N*-dimensional column vector *y* of measured values is the **error** (which is minimised with the least squares method).
  
  If you have *q* dependent variables *yi* (*i = 1...q*), each of them would generally have its own set of weights *beta*. In other words, the relation between the input variables and *each* *yi* will be *different*. The dependent variables can also be arranged in a matrix (just like the input matrix), and its dimensionality will be *N x q*, where *q*. In that case, the *beta* matrix will **not be a single *N*-dimensional column vector but an *N x q* matrix**. Each column in the *beta* matrix gives the relation between the input variables and the *q*th dependent variable *yq*. In that respect, a single observation of all variables *yi* will be a *row* vector in the *y* matrix.
  
  I hope that this clarifies things up a bit. In summary, the explanation in the textbook is correct, but the notation is hard to follow and at times plain misleading (as in the case of variable and component at the beginning of the section). Honestly, you can get a much more intuitive explanation of linear regression from the [Wikipedia](https://en.wikipedia.org/wiki/Linear_regression) article.
  
  
  ---
  
  