Body: <p>I have a data file from a Monte Carlo simulation of fifteen protein chains. The file contains 10 million <code>r_end_to_end</code> 3D vectors as rows and 3 x 15 = 45 columns.</p>
  <p>My task is to find the exact row where the simulation stabilized. In other words, I need to find autocorrelation time. In other words, I need to find the <code>Tau0</code> time.</p>
  <p>Now, the problem is that the data file is huge.</p>
  <p>So, reading the entire 10 million rows is not practical for frequent testing, plot generation, curve-fitting, etc.</p>
  <p>Is there any technique that I can use to achieve the required computation without reading all 10 million rows?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-07T14:25:46.663Z
Comments.Text: I don't think we can help without seeing an example input and the expected output. Of course I understand you can't show us 10 million lines, but could you give a toy example so we can understand the kind of data you are working with? And how slow? I made a toy example file with 10 million rows and 45 columns with `perl -le 'for my $i (1..10000000){ print join("\t", (1..45));  }' > file`. A simple operation (`awk '($2 > $3){k++}' file`) on the resulting 1.2G file only took ~1 second. Can you explain the kind of operations you will be running?
Comments.UserDisplayName: terdon
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-08T08:11:09.840Z
Comments.Text: For the frequent data access stay away from CSV/TSV text formats and switch to i.e `parquet` or `numpy arrays`.
Comments.UserDisplayName: darked89
Tags: <data-preprocessing>
Title: Is there any technique that I can use to achieve the required computation without reading all 10 million rows?
ThreadText: # Is there any technique that I can use to achieve the required computation without reading all 10 million rows?
  
  (Asked by: username_0 on 2024-06-05)
  
  I have a data file from a Monte Carlo simulation of fifteen protein chains. The file contains 10 million `r_end_to_end` 3D vectors as rows and 3 x 15 = 45 columns.
  
  My task is to find the exact row where the simulation stabilized. In other words, I need to find autocorrelation time. In other words, I need to find the `Tau0` time.
  
  Now, the problem is that the data file is huge.
  
  So, reading the entire 10 million rows is not practical for frequent testing, plot generation, curve-fitting, etc.
  
  Is there any technique that I can use to achieve the required computation without reading all 10 million rows?
  
  **Tags:** data-preprocessing
  
  > **Comments:**
  > **username_1**: I don't think we can help without seeing an example input and the expected output. Of course I understand you can't show us 10 million lines, but could you give a toy example so we can understand the kind of data you are working with? And how slow? I made a toy example file with 10 million rows and 45 columns with `perl -le 'for my $i (1..10000000){ print join("\t", (1..45));  }' > file`. A simple operation (`awk '($2 > $3){k++}' file`) on the resulting 1.2G file only took ~1 second. Can you explain the kind of operations you will be running?  
  > **username_2**: For the frequent data access stay away from CSV/TSV text formats and switch to i.e `parquet` or `numpy arrays`.  
  
  ---
  
  ## 0 answers
  
  