Answers.Body: <p>This isn't typically due to a flaw in the dataset but rather reflects the structural overlap in predictor information within the data. When predictors are nearly collinear the matrix of predictor values becomes nearly singular, thus it becomes difficult to pinpoint unique contributions of each predictor. During estimation the linear regression procedure seeks a best fit but struggles to allocate weight accurately between highly correlated variables. As a result small changes in data with limited samples could possibly produce large shifts in coefficients as the model attempts to settle on new estimates.</p>
  <p>Here's a concrete example written in python.</p>
  <pre><code>import pandas as pd
  from sklearn.linear_model import LinearRegression
  
  # Initial data setup
  data = {
      'SquareFootage': [1500, 2000, 2500, 3000],
      'Rooms': [5, 7, 8, 9],
      'Price': [200000, 280000, 340000, 400000]
  }
  
  # Create DataFrame
  df = pd.DataFrame(data)
  
  # Function to fit model and print coefficients
  def fit_model(dataframe):
      X = dataframe[['SquareFootage', 'Rooms']]
      y = dataframe['Price']
      model = LinearRegression()
      model.fit(X, y)
      return model.coef_
  
  # Initial fit with original data
  print(&quot;Original Coefficients:&quot;)
  original_coeffs = fit_model(df)
  print(f&quot;Square Footage: {original_coeffs[0]:.2f}, Rooms: {original_coeffs[1]:.2f}&quot;)
  
  # Make a small change to the data (decrease Square Footage for House C a bit)
  df_modified = df.copy()
  df_modified.loc[2, 'SquareFootage'] = 2400
  
  # Fit model with modified data
  print(&quot;\nModified Coefficients (after small data change):&quot;)
  modified_coeffs = fit_model(df_modified)
  print(f&quot;Square Footage: {modified_coeffs[0]:.2f}, Rooms: {modified_coeffs[1]:.2f}&quot;)
  
  </code></pre>
  <p>After running the program you get below results for comparison where the weights are sensitive :</p>
  <pre><code>Original Coefficients:
  Square Footage: 80.00, Rooms: 20000.00
  
  Modified Coefficients (after small data change):
  Square Footage: 65.98, Rooms: 25773.20
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-29T04:54:21.637Z
Answers.LastActivityDate: 2024-10-29T05:13:58.903Z
Answers.LastEditDate: 2024-10-29T05:13:58.903Z
Answers.UserDisplayName: cinch
Answers.Body: <p>You can start to see this in a fairly straightforward simulation using a linear regression. Simulate 1000 linear regressions with highly correlated features and 1000 with uncorrelated features. Which one has a wider range of fitted coefficients? How does that range depend on the extent of the correlation?</p>
  <p>The underlying math is that the multicollinearity inflates the coefficient standard errors through the so-called <a href="https://en.m.wikipedia.org/wiki/Variance_inflation_factor" rel="nofollow noreferrer"><em>variance inflation factor</em></a>.</p>
  <p>I don’t have access to R until tomorrow night, but something like the code below should do the trick.</p>
  <pre><code>library(MASS)
  set.seed(2024)
  N &lt;- 100
  R &lt;- 1000
  b_corr &lt;- b_ind &lt;- rep(NA, R)
  for (i in 1:R){
  
  X_corr &lt;- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.99, 0.99, 1), 2, 2)
  X_ind &lt;- cbind(X_corr[, 1], sample(X_corr[, 2], N, replace = F)
  
  e &lt;- rnorm(N)
  y_corr &lt;- X_corr %*% c(1, 1) + e
  y_ind &lt;- X_ind %*% c(1, 1) + 
  
  L_corr &lt;- lm(y_corr ~ X_corr)
  L_ind &lt;- lm(y_ind ~ X_ind)
  
  b_corr[i] &lt;- coef(L_corr)[2]
  b_ind[i] &lt;- coef(L_ind)[2]
  }
  var(b_corr)
  var(b_ind)
  </code></pre>
  <p>(Without access to R, I can’t guarantee that the above code will compile, but it gives the gist of how to simulate this.)</p>
  <p>How much larger is the variance with highly correlated features (one form of feature multicollinearity) vs the variance when the features are independent yet everything else is the same?</p>
  <p>If you want to see this from real data, perhaps try taking various subsets from a dataset with feature multicollinearity and watch how the coefficients change as the make changes to the training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-01T13:53:56.767Z
Answers.Text: Yeah, there is nice answer. So, I would like to add a little bit that we are not calculating b_corr and b_ind, but m2_corr and m2_ind actually (at least as I was able to translate into python this is the case, otherwise we need to take intercept)
Answers.UserDisplayName: Exfell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-01T16:50:45.207Z
Answers.Text: @Exfell What is `m2`?
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T18:48:42.477Z
Answers.Text: I got it. The explanation I saw had the notation y = m1x1+m2x2+b. You probably meant the letter beta as a coefficient, right? It's just that then we still need to add some index, since we have two coefficients in this model, and we take only one (the result will be the same, just for design purposes).
Answers.UserDisplayName: Exfell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T18:50:50.100Z
Answers.Text: So in the comment I meant to change the name from intercept (b) to m2 (as I understood you took the second coefficient in the code)
Answers.UserDisplayName: Exfell
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-02T20:56:52.983Z
Answers.Text: My simulation calculates one intercept and two slopes. However, it only stores one slope parameter. You can tweak the simulation to store all three regression parameters and look at how their variances are affected by the correlation.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-28T13:17:18.967Z
Answers.LastActivityDate: 2024-09-28T13:17:18.967Z
Answers.UserDisplayName: Dave
Body: <p>I've heard that the coefficients in multicollinearity are very sensitive, and can change due to small changes in the data.... Isn't it a problem with the dataset itself that we have different data? And can you give me an (realistic) example where the coefficients directly change very much from small changes in the data?</p>
  
Tags: <features>
Title: What is the sensitivity of coefficients in multicollinearity?
UserDisplayName: Exfell
ThreadText: # What is the sensitivity of coefficients in multicollinearity?
  
  (Asked by: username_0 on 2024-09-28)
  
  I've heard that the coefficients in multicollinearity are very sensitive, and can change due to small changes in the data.... Isn't it a problem with the dataset itself that we have different data? And can you give me an (realistic) example where the coefficients directly change very much from small changes in the data?
  
  **Tags:** features
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-09-28. Score: 2 | Accepted answer
  
  You can start to see this in a fairly straightforward simulation using a linear regression. Simulate 1000 linear regressions with highly correlated features and 1000 with uncorrelated features. Which one has a wider range of fitted coefficients? How does that range depend on the extent of the correlation?
  
  The underlying math is that the multicollinearity inflates the coefficient standard errors through the so-called [*variance inflation factor*](https://en.m.wikipedia.org/wiki/Variance_inflation_factor).
  
  I don’t have access to R until tomorrow night, but something like the code below should do the trick.
  
  ```
  library(MASS)
  set.seed(2024)
  N <- 100
  R <- 1000
  b_corr <- b_ind <- rep(NA, R)
  for (i in 1:R){
  
  X_corr <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.99, 0.99, 1), 2, 2)
  X_ind <- cbind(X_corr[, 1], sample(X_corr[, 2], N, replace = F)
  
  e <- rnorm(N)
  y_corr <- X_corr %*% c(1, 1) + e
  y_ind <- X_ind %*% c(1, 1) + 
  
  L_corr <- lm(y_corr ~ X_corr)
  L_ind <- lm(y_ind ~ X_ind)
  
  b_corr[i] <- coef(L_corr)[2]
  b_ind[i] <- coef(L_ind)[2]
  }
  var(b_corr)
  var(b_ind)
  
  ```
  
  (Without access to R, I can’t guarantee that the above code will compile, but it gives the gist of how to simulate this.)
  
  How much larger is the variance with highly correlated features (one form of feature multicollinearity) vs the variance when the features are independent yet everything else is the same?
  
  If you want to see this from real data, perhaps try taking various subsets from a dataset with feature multicollinearity and watch how the coefficients change as the make changes to the training data.
  
  > **Comments:**
  > **username_0**: Yeah, there is nice answer. So, I would like to add a little bit that we are not calculating b_corr and b_ind, but m2_corr and m2_ind actually (at least as I was able to translate into python this is the case, otherwise we need to take intercept)  
  > **username_2**: @username_0 What is `m2`?  
  > **username_0**: I got it. The explanation I saw had the notation y = m1x1+m2x2+b. You probably meant the letter beta as a coefficient, right? It's just that then we still need to add some index, since we have two coefficients in this model, and we take only one (the result will be the same, just for design purposes).  
  > **username_0**: So in the comment I meant to change the name from intercept (b) to m2 (as I understood you took the second coefficient in the code)  
  > **username_2**: My simulation calculates one intercept and two slopes. However, it only stores one slope parameter. You can tweak the simulation to store all three regression parameters and look at how their variances are affected by the correlation.  
  
  ---
  
  ### Answer by username_1 on 2024-10-29. Score: 1
  
  This isn't typically due to a flaw in the dataset but rather reflects the structural overlap in predictor information within the data. When predictors are nearly collinear the matrix of predictor values becomes nearly singular, thus it becomes difficult to pinpoint unique contributions of each predictor. During estimation the linear regression procedure seeks a best fit but struggles to allocate weight accurately between highly correlated variables. As a result small changes in data with limited samples could possibly produce large shifts in coefficients as the model attempts to settle on new estimates.
  
  Here's a concrete example written in python.
  
  ```
  import pandas as pd
  from sklearn.linear_model import LinearRegression
  
  # Initial data setup
  data = {
      'SquareFootage': [1500, 2000, 2500, 3000],
      'Rooms': [5, 7, 8, 9],
      'Price': [200000, 280000, 340000, 400000]
  }
  
  # Create DataFrame
  df = pd.DataFrame(data)
  
  # Function to fit model and print coefficients
  def fit_model(dataframe):
      X = dataframe[['SquareFootage', 'Rooms']]
      y = dataframe['Price']
      model = LinearRegression()
      model.fit(X, y)
      return model.coef_
  
  # Initial fit with original data
  print("Original Coefficients:")
  original_coeffs = fit_model(df)
  print(f"Square Footage: {original_coeffs[0]:.2f}, Rooms: {original_coeffs[1]:.2f}")
  
  # Make a small change to the data (decrease Square Footage for House C a bit)
  df_modified = df.copy()
  df_modified.loc[2, 'SquareFootage'] = 2400
  
  # Fit model with modified data
  print("\nModified Coefficients (after small data change):")
  modified_coeffs = fit_model(df_modified)
  print(f"Square Footage: {modified_coeffs[0]:.2f}, Rooms: {modified_coeffs[1]:.2f}")
  
  ```
  
  After running the program you get below results for comparison where the weights are sensitive :
  
  ```
  Original Coefficients:
  Square Footage: 80.00, Rooms: 20000.00
  
  Modified Coefficients (after small data change):
  Square Footage: 65.98, Rooms: 25773.20
  
  ```
  
  
  ---
  
  