Answers.Body: <p>You pandas flatten sounds right to me. And unlike psychological stock price, TEC ionospheric physical data noise are more white and you certainly have more regularity as your advantage to more accurately forecast ionospheric movement than in general. Empirically the trial behavior you're observing after shuffling of your flattened time series data within all 3-day windows, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent structure of your data with regularity.</p>
  <p>The autoregressive AR(3) model should excel at your problem due to the exposed strong temporal dependency, while the cross-sectional linear regression model fails badly when shuffling features breaks such temporal order. Therefore when you originally apply linear regression model without shuffling, you get abnormally good result in the sense of very small average MSE as it's essentially equivalent to the AR(3) model. Further since cross-sectional models won't propagate errors like AR models, linear regression model may be even better to predict any days ahead directly for your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-27T22:34:24.773Z
Answers.Text: Would tell me my code is right? I request you, kindly please review my code. If I get more satisfactory answer, I will award 100+ for existing answer.
Answers.UserDisplayName: S. M.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T09:06:25.560Z
Answers.Text: @S.M. I’ve checked the critical flattern part, so far so good as already expressed in my answer. If you can share your raw csv data which includes both training and test dataset from your code, I can run and check further.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T10:30:40.873Z
Answers.Text: In question, below link I have provided compressed csv filé links for all the available day.
Answers.UserDisplayName: S. M.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-28T14:44:00.770Z
Answers.Text: I need your opinion, please let me know if you find the file or not.
Answers.UserDisplayName: S. M.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T05:59:34.417Z
Answers.Text: I've run the  code in your OP with your 234 csv data files, also I plot the RMSE which makes more sense when comparing residues with labels than MSE, they are all on the range of 5e-14 but the shape is different from your OP's plot, mine is more volatile with more visible peaks and valleys in a scaled output plot. Again I think for tec data there's much regularity like the near-term weather forecast, LR model could predict with high accuracy easily here with enough data like yours. Hope this helps. You could also ask in data science exchange where this type of code review question is more fit.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T06:36:33.897Z
Answers.Text: Also bear in mind the (R)MSE here is the average for each data point's residue out of the daily 600 total data points.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T06:44:29.980Z
Answers.Text: You mean, my code is in right track?
Answers.UserDisplayName: S. M.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T06:48:12.530Z
Answers.Text: Yes from my perspective, since here it's essentially a multivariate lr model with shapes as design matrix XX[90, 1800], labels yy[90,60] after your flattening.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T06:51:17.583Z
Answers.Text: As you have written in above comment, yy should be yy[90, 600] not yy[90,60].
Answers.UserDisplayName: S. M.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T07:04:55.317Z
Answers.Text: sorry, you're correct, just typo. If you know structural equation modeling (SEM) in causal statistical inference, tec data here suggests all data points may follow some simple deterministic linear structural equations with some small white noise.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-23T08:59:32.240Z
Answers.LastActivityDate: 2024-11-23T08:59:32.240Z
Answers.UserDisplayName: cinch
Body: <p>I have the following task to do: I have time series data. Training by the consecutive 3 days to predict the each 4th day. Each day data represents one CSV file which has dimension 24x25. Every datapoints of each CSV file are pixels.</p>
  <p>Now, I need to do that, predict day4(means 4th day) by using training data day1, day2, day3(means three consecutive days), after then calculate mse between predicted day4 data and original day4 data. Let's call it mse1.</p>
  <p>Similarly, I need to predict the day5 (means 5th day) by using training data day2, day3, day4, and then calculate the mse2(mse between predicted day5 data and original day5 data)</p>
  <p>I need to predict day6(means 6th day)by using training data day3, day4, day5,  and then calculate mse3(mse between predicted day6 data and original day6)</p>
  <p>..........</p>
  <p>And finally I want to Predict day93 by using training data day90, day91, day92,calculate mse90(mse between predicted day93 data and original day93)</p>
  <p>I want to use in this case , <a href="https://en.m.wikipedia.org/wiki/Linear_regression" rel="nofollow noreferrer">Linear regression</a>,and we have 90 mse for this model.</p>
  <pre><code>import os
  import pandas as pd
  import numpy as np
  from sklearn.linear_model import LinearRegression
  from sklearn.preprocessing import MinMaxScaler
  import matplotlib.pyplot as plt
  
  # Paths
  data_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\All_data'
  output_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\90_days_merged'
  
  # Ensure the output folder exists
  os.makedirs(output_folder, exist_ok=True)
  
  # List all CSV files in the folder
  csv_files = [f for f in os.listdir(data_folder) if f.endswith('.csv')]
  
  # Sort the files based on the numeric part extracted from the filename
  csv_files = sorted(csv_files, key=lambda x: int(x.split('_Day')[1].split('_')[0]))
  
  # Prepare data
  data_list = [pd.read_csv(os.path.join(data_folder, file), header=None).values for file in csv_files]
  data_array = np.array(data_list)  # Shape: (num_days, 24, 25)
  
  # Flatten the data for easier handling in regression models
  num_days, rows, cols = data_array.shape
  data_flattened = data_array.reshape(num_days, -1)  # Shape: (num_days, 600)
  
  # Prepare features and target matrix for range (3, num_days)
  X = np.array([data_flattened[i-3:i].flatten() for i in range(3, num_days)])  # Shape: (num_days-3, 1800)
  y = data_flattened[3:num_days]  # Target is the 4th day in each sequence
  
  # Train-Test Split and Validation (Separate fixed split)
  train_size = int(0.8 * len(X))  # 80% for training
  X_train = X[:train_size]
  y_train = y[:train_size]
  X_test = X[train_size:]
  y_test = y[train_size:]
  
  # Scaling the data
  scaler_X = MinMaxScaler()
  scaler_X.fit(X_train)  # Fit on training set
  X_train_scaled = scaler_X.transform(X_train)
  X_test_scaled = scaler_X.transform(X_test)
  
  scaler_y = MinMaxScaler()
  scaler_y.fit(y_train)  # Fit on training set
  y_train_scaled = scaler_y.transform(y_train)
  y_test_scaled = scaler_y.transform(y_test)
  
  ### Linear Regression
  lr_model = LinearRegression()
  lr_model.fit(X_train_scaled, y_train_scaled)
  
  y_pred_test_scaled_lr = lr_model.predict(X_test_scaled)
  y_pred_test_lr = scaler_y.inverse_transform(y_pred_test_scaled_lr)
  
  # Validation for Days 3 to 93
  XX = np.array([data_flattened[i-3:i].flatten() for i in range(3, 96)])  # Shape: (90, 1800)
  yy = data_flattened[3:93]  # Target for validation
  
  yy_pred_lr = lr_model.predict(scaler_X.transform(XX))
  yy_pred_lr = scaler_y.inverse_transform(yy_pred_lr)
  
  # Calculate residuals for Linear Regression
  residuals_lr = [np.mean((yy[i] - yy_pred_lr[i])**2) for i in range(len(yy))]
  
  # Plot residuals for all models
  days = [f'Day {i+1}' for i in range(90)]  # Start labels from Day 4 to Day 93
  plt.figure(figsize=(12, 6))
  plt.plot(days, residuals_lr, label='Linear Regression Residuals', marker='o')
  
  # Configure plot
  plt.xticks(ticks=range(0, len(days), 2), labels=[f'Day {i+1}' for i in range(0, len(days), 2)], rotation=45, ha='right')
  plt.xlabel('Days (Validation Set)')
  plt.ylabel('Residuals (MSE)')
  plt.title('Residuals for Models (Validation Set)')
  plt.legend()
  plt.grid(True)
  
  # Save and show plot
  plt.savefig(os.path.join(output_folder, 'residuals_plot_models_comparison_with_naive.png'))
  plt.show()
  </code></pre>
  <p>My result:
  <a href="https://i.sstatic.net/x3s3O1iI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/x3s3O1iI.png" alt="enter image description here" /></a></p>
  <p>We know that linear regression models often do not do very well with time series data because the assumption of independent and identically distributed data is usually violated.</p>
  <p>But in my case from the above plot, regression model is doing exceptionally very well(means <code>mean squared error</code> are very low, closes to zero), would anybody check all the my regression model inside the code(if I made any mistakes or bugs that I might not be aware of.)?</p>
  <p>My all 93 days data folder <a href="https://drive.google.com/file/d/1S_J9kuPlh9zDoO_1tg32ZDTwGaOtcgh-/view?usp=sharing" rel="nofollow noreferrer">link</a> that I used for code.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T00:53:16.487Z
Comments.Text: @cinch it's [tec](https://impc.dlr.de/products/total-electron-content/forecast-tec/one-hour-forecast-tec-map-global) data, each data points are pixels. So, would you write the code review?
Comments.UserDisplayName: S. M.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T01:43:24.100Z
Comments.Text: @cinch for any day we have only 15 mins data available, this data is 24×25 csv filé. On which day the data is available, it's 15 mins data.
Comments.UserDisplayName: S. M.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T04:12:11.483Z
Comments.Text: @cinch but after shuffling lr model is working worst as expected, but lstm model is started working well than lr model after shuffling. what you comment on this scenario? since we know that in low amount of data( in my case 3-day window), the sophisticated model like lstm is often work worst, but after your suggesting shuffle, lr model is working as expected but lstm model not working as expected. what would you comment on this?
Comments.UserDisplayName: S. M.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T06:30:38.400Z
Comments.Text: @cinch would I share with you complete code with lstm?
Comments.UserDisplayName: S. M.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T07:40:02.420Z
Comments.Text: This question is about lr model not lstm. lstm starts working better than lr for all 3-day window test samples doesn't necessary mean lstm is not working as expected since it may still learn local patterns that persist even in the absence of strict time-based structure, though lstm generally needs much more data and certainly is not recommended for your 3-day prediction problem. The behavior you're observing after shuffling, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent regular structure of your data.
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T08:36:52.327Z
Comments.Text: @cinch thank you for your valuable comments. I have one little confusion, without shuffle ridge regression is working bad than linear but after shuffle I get the reverse, what would you comments on this?
Comments.UserDisplayName: S. M.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-23T08:41:53.983Z
Comments.Text: Sure. Depending on the sense of 'bad', it may be statistical flux without significance, generally with 1800 features you need some regularization.
Comments.UserDisplayName: cinch
Tags: <python><tensorflow><regression><features><linear-regression>
Title: Regression model is doing exceptionally very well on time series
UserDisplayName: S. M.
ThreadText: # Regression model is doing exceptionally very well on time series
  
  (Asked by: username_0 on 2024-11-22)
  
  I have the following task to do: I have time series data. Training by the consecutive 3 days to predict the each 4th day. Each day data represents one CSV file which has dimension 24x25. Every datapoints of each CSV file are pixels.
  
  Now, I need to do that, predict day4(means 4th day) by using training data day1, day2, day3(means three consecutive days), after then calculate mse between predicted day4 data and original day4 data. Let's call it mse1.
  
  Similarly, I need to predict the day5 (means 5th day) by using training data day2, day3, day4, and then calculate the mse2(mse between predicted day5 data and original day5 data)
  
  I need to predict day6(means 6th day)by using training data day3, day4, day5, and then calculate mse3(mse between predicted day6 data and original day6)
  
  ..........
  
  And finally I want to Predict day93 by using training data day90, day91, day92,calculate mse90(mse between predicted day93 data and original day93)
  
  I want to use in this case , [Linear regression](https://en.m.wikipedia.org/wiki/Linear_regression),and we have 90 mse for this model.
  
  ```
  import os
  import pandas as pd
  import numpy as np
  from sklearn.linear_model import LinearRegression
  from sklearn.preprocessing import MinMaxScaler
  import matplotlib.pyplot as plt
  
  # Paths
  data_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\All_data'
  output_folder = r'C:\Users\alokj\OneDrive\Desktop\jupyter_proj\90_days_merged'
  
  # Ensure the output folder exists
  os.makedirs(output_folder, exist_ok=True)
  
  # List all CSV files in the folder
  csv_files = [f for f in os.listdir(data_folder) if f.endswith('.csv')]
  
  # Sort the files based on the numeric part extracted from the filename
  csv_files = sorted(csv_files, key=lambda x: int(x.split('_Day')[1].split('_')[0]))
  
  # Prepare data
  data_list = [pd.read_csv(os.path.join(data_folder, file), header=None).values for file in csv_files]
  data_array = np.array(data_list)  # Shape: (num_days, 24, 25)
  
  # Flatten the data for easier handling in regression models
  num_days, rows, cols = data_array.shape
  data_flattened = data_array.reshape(num_days, -1)  # Shape: (num_days, 600)
  
  # Prepare features and target matrix for range (3, num_days)
  X = np.array([data_flattened[i-3:i].flatten() for i in range(3, num_days)])  # Shape: (num_days-3, 1800)
  y = data_flattened[3:num_days]  # Target is the 4th day in each sequence
  
  # Train-Test Split and Validation (Separate fixed split)
  train_size = int(0.8 * len(X))  # 80% for training
  X_train = X[:train_size]
  y_train = y[:train_size]
  X_test = X[train_size:]
  y_test = y[train_size:]
  
  # Scaling the data
  scaler_X = MinMaxScaler()
  scaler_X.fit(X_train)  # Fit on training set
  X_train_scaled = scaler_X.transform(X_train)
  X_test_scaled = scaler_X.transform(X_test)
  
  scaler_y = MinMaxScaler()
  scaler_y.fit(y_train)  # Fit on training set
  y_train_scaled = scaler_y.transform(y_train)
  y_test_scaled = scaler_y.transform(y_test)
  
  ### Linear Regression
  lr_model = LinearRegression()
  lr_model.fit(X_train_scaled, y_train_scaled)
  
  y_pred_test_scaled_lr = lr_model.predict(X_test_scaled)
  y_pred_test_lr = scaler_y.inverse_transform(y_pred_test_scaled_lr)
  
  # Validation for Days 3 to 93
  XX = np.array([data_flattened[i-3:i].flatten() for i in range(3, 96)])  # Shape: (90, 1800)
  yy = data_flattened[3:93]  # Target for validation
  
  yy_pred_lr = lr_model.predict(scaler_X.transform(XX))
  yy_pred_lr = scaler_y.inverse_transform(yy_pred_lr)
  
  # Calculate residuals for Linear Regression
  residuals_lr = [np.mean((yy[i] - yy_pred_lr[i])**2) for i in range(len(yy))]
  
  # Plot residuals for all models
  days = [f'Day {i+1}' for i in range(90)]  # Start labels from Day 4 to Day 93
  plt.figure(figsize=(12, 6))
  plt.plot(days, residuals_lr, label='Linear Regression Residuals', marker='o')
  
  # Configure plot
  plt.xticks(ticks=range(0, len(days), 2), labels=[f'Day {i+1}' for i in range(0, len(days), 2)], rotation=45, ha='right')
  plt.xlabel('Days (Validation Set)')
  plt.ylabel('Residuals (MSE)')
  plt.title('Residuals for Models (Validation Set)')
  plt.legend()
  plt.grid(True)
  
  # Save and show plot
  plt.savefig(os.path.join(output_folder, 'residuals_plot_models_comparison_with_naive.png'))
  plt.show()
  
  ```
  
  My result:
  [![enter image description here](https://i.sstatic.net/x3s3O1iI.png)](https://i.sstatic.net/x3s3O1iI.png)
  
  We know that linear regression models often do not do very well with time series data because the assumption of independent and identically distributed data is usually violated.
  
  But in my case from the above plot, regression model is doing exceptionally very well(means `mean squared error` are very low, closes to zero), would anybody check all the my regression model inside the code(if I made any mistakes or bugs that I might not be aware of.)?
  
  My all 93 days data folder [link](https://drive.google.com/file/d/1S_J9kuPlh9zDoO_1tg32ZDTwGaOtcgh-/view?usp=sharing) that I used for code.
  
  **Tags:** python, tensorflow, regression, features, linear-regression
  
  > **Comments:**
  > **username_0**: @username_1 it's [tec](https://impc.dlr.de/products/total-electron-content/forecast-tec/one-hour-forecast-tec-map-global) data, each data points are pixels. So, would you write the code review?  
  > **username_0**: @username_1 for any day we have only 15 mins data available, this data is 24×25 csv filé. On which day the data is available, it's 15 mins data.  
  > **username_0**: @username_1 but after shuffling lr model is working worst as expected, but lstm model is started working well than lr model after shuffling. what you comment on this scenario? since we know that in low amount of data( in my case 3-day window), the sophisticated model like lstm is often work worst, but after your suggesting shuffle, lr model is working as expected but lstm model not working as expected. what would you comment on this?  
  > **username_0**: @username_1 would I share with you complete code with lstm?  
  > **username_1**: This question is about lr model not lstm. lstm starts working better than lr for all 3-day window test samples doesn't necessary mean lstm is not working as expected since it may still learn local patterns that persist even in the absence of strict time-based structure, though lstm generally needs much more data and certainly is not recommended for your 3-day prediction problem. The behavior you're observing after shuffling, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent regular structure of your data.  
  > **username_0**: @username_1 thank you for your valuable comments. I have one little confusion, without shuffle ridge regression is working bad than linear but after shuffle I get the reverse, what would you comments on this?  
  > **username_1**: Sure. Depending on the sense of 'bad', it may be statistical flux without significance, generally with 1800 features you need some regularization.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-23. Score: 1 | Accepted answer
  
  You pandas flatten sounds right to me. And unlike psychological stock price, TEC ionospheric physical data noise are more white and you certainly have more regularity as your advantage to more accurately forecast ionospheric movement than in general. Empirically the trial behavior you're observing after shuffling of your flattened time series data within all 3-day windows, if the new MSE is much larger than the original MSE, suggests that shuffling exposes the intrinsic low noise temporally-dependent structure of your data with regularity.
  
  The autoregressive AR(3) model should excel at your problem due to the exposed strong temporal dependency, while the cross-sectional linear regression model fails badly when shuffling features breaks such temporal order. Therefore when you originally apply linear regression model without shuffling, you get abnormally good result in the sense of very small average MSE as it's essentially equivalent to the AR(3) model. Further since cross-sectional models won't propagate errors like AR models, linear regression model may be even better to predict any days ahead directly for your problem.
  
  > **Comments:**
  > **username_0**: Would tell me my code is right? I request you, kindly please review my code. If I get more satisfactory answer, I will award 100+ for existing answer.  
  > **username_1**: @S.M. I’ve checked the critical flattern part, so far so good as already expressed in my answer. If you can share your raw csv data which includes both training and test dataset from your code, I can run and check further.  
  > **username_0**: In question, below link I have provided compressed csv filé links for all the available day.  
  > **username_0**: I need your opinion, please let me know if you find the file or not.  
  > **username_1**: I've run the  code in your OP with your 234 csv data files, also I plot the RMSE which makes more sense when comparing residues with labels than MSE, they are all on the range of 5e-14 but the shape is different from your OP's plot, mine is more volatile with more visible peaks and valleys in a scaled output plot. Again I think for tec data there's much regularity like the near-term weather forecast, LR model could predict with high accuracy easily here with enough data like yours. Hope this helps. You could also ask in data science exchange where this type of code review question is more fit.  
  > **username_1**: Also bear in mind the (R)MSE here is the average for each data point's residue out of the daily 600 total data points.  
  > **username_0**: You mean, my code is in right track?  
  > **username_1**: Yes from my perspective, since here it's essentially a multivariate lr model with shapes as design matrix XX[90, 1800], labels yy[90,60] after your flattening.  
  > **username_0**: As you have written in above comment, yy should be yy[90, 600] not yy[90,60].  
  > **username_1**: sorry, you're correct, just typo. If you know structural equation modeling (SEM) in causal statistical inference, tec data here suggests all data points may follow some simple deterministic linear structural equations with some small white noise.  
  
  ---
  
  