title: Fastest way to count lines in file in MATLAB (Perl faster than C?)
body: <p>I have to count the number of lines in 10+ GB csv files in my research. The classic way of doing this on MATLAB seems to be using <code>textscan()</code> with <code>\n</code> as the delimiter, but that has a huge memory footprint and is awfully slow. I was advised to write a Perl script and call it using <code>str2double(perl('countlines.pl', path))</code>, which does seem significantly faster:</p>
  
  <pre><code># countlines.pl
  while (&lt;&gt;) {};
  print $.,"\n";
  </code></pre>
  
  <p>Then, I wanted to see if I had any advantage by writing a MEX function that does the same in C, but had no luck, and more surprisingly, I found that this was about 10x slower than the Perl script (using the LLVM compiler on Xcode 4.6.3):</p>
  
  <pre><code>//countlines.c
  
  #include "mex.h"
  
  void countlines(char *filepath, double *numLines)
  {
      /* Routine */
  
      numLines[0] = 0;
      FILE *inputFile = fopen(filepath, "r");
      int ch;
  
      while (EOF != (ch=getc(inputFile)))
          if ('\n' == ch)
              ++numLines[0];
  }
  
  void mexFunction( int nlhs, mxArray *plhs[],
                    int nrhs, const mxArray *prhs[])
  {
      /* Gateway function */
  
      int bufferLength, status;
      char *filepath;                 // Input: File path
      double *numLines;               // Output Number of lines
  
      bufferLength = (mxGetM(prhs[0]) * mxGetN(prhs[0])) + 1; // Get length of string
      filepath = mxCalloc(bufferLength, sizeof(char)); // Allocate memory for input
  
      // Copy the string data from prhs[0] into a C string
      status = mxGetString(prhs[0], filepath, bufferLength);
      if (status != 0)
          mexErrMsgIdAndTxt("utils:countlines:insufficientSpace", "Insufficient space, string is truncated.");
  
      // Create the output matrix and get a pointer to the real data in the output matrix
      plhs[0] = mxCreateDoubleMatrix(1,(mwSize)1,mxREAL);
      numLines = mxGetPr(plhs[0]);
  
      // Call the C routine
      countlines(filepath, numLines);
  }
  </code></pre>
  
  <p>So,</p>
  
  <ol>
  <li>Where is this overhead coming from in the MEX function besides the gateway function?</li>
  <li>Is there anything else I can do to make this faster? I'm open to using any language so long as we can get the routine to interface with MATLAB. It seems the only other approach is to memory-map chunks of the file and split the workload across a few cores.</li>
  </ol>