title: Speed and Memory managment of C vs Perl
body: <p>Context:
  Currently, my team has a perl script that does many things, one of which is that it stores 1 byte hex-values in a hashmap(array yields similar results). The input data can range from hundreds of MB to 10s of GBs. At the moment, when we run a 1GB input (1 billion entries), the script takes about 10 minutes and then errors out due to using all 16GB of my RAM. I'm told a 1GB input can expand to nearly 300GB.</p>
  <p>We then made a comparable C program, and found that it takes a few minutes and uses only 1.1GBs.</p>
  <p>I wrote the code below in order to simply test how C and Perl perform for writing 1 Billion values. I'm finding that the Perl code takes around 186seconds and &gt;70GB of memory to run, while the C code only takes two seconds and 1GB. I use time and memusage to determine measurements.</p>
  <p>Question:
  Is perl actually this slow and bad at memory management, or am I missing something. The literature i've read online, has the Perl should be slower due to the flexibility that it provides, but relatively not super slow since it is written in C.</p>
  <p>Perl code example of memory usage:</p>
  <pre><code>use strict;
  
  my @list;
  
  for(my $a=0; $a &lt; 1000000000;$a++){
      $list[$a]=1; # 1 is just to simulate some data.
  }
  print 'done';
  </code></pre>
  <p>C code:</p>
  <pre><code>#include &lt;stdlib.h&gt;
  
  int main() {
  
    int size = 1000000000;
      unsigned char *data = (unsigned char*)malloc(size*sizeof(unsigned char));
  
      unsigned char byte = 'a';
      int address = 0;
      while (address &lt; size) {                                                                                                                  
        data[address]=byte;
        address++;
  
      }
      printf(&quot;done %i.\n&quot;,address);
  
      return 0;
  }
  
  
  </code></pre>
  <p>Also tried in python, which was worse than perl in terms of speed.</p>
  <pre><code>data = []
  
  d = format(231,'#04x')
  
  while address &lt; 1000000000:
      data.append(d)
      address += 1
  print &quot;done&quot;
  while(1):
      continue
  </code></pre>
  <p>note: I haven't used a profiler yet since the evaluation code is simple.</p>
  <p>Because of these performance issues, I found a solution where called SWIG which allows me to wrap C code and run it in perl; however, I have some follow up questions about it. :)</p>
  <p>edit: tag</p>