Stack Exchange
perl error "out of memory" glibc detected - how to fix
I got this crazy message:
Out of memory!
*** glibc detected *** perl: double free or corruption (!prev): 0x00000003f8f89690 ***
======= Backtrace: =========
/lib64/libc.so.6[0x372bc75916]
/lib64/libc.so.6[0x372bc78443]
/usr/lib64/perl5/CORE/libperl.so(Perl_pregfree+0xd1)[0x3c5da74581]
/usr/lib64/perl5/CORE/libperl.so(Perl_op_clear+0xeb)[0x3c5da3829b]
/usr/lib64/perl5/CORE/libperl.so(Perl_op_free+0x15c)[0x3c5da3670c]
/usr/lib64/perl5/CORE/libperl.so(Perl_op_free+0xbc)[0x3c5da3666c]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 2229974                            /usr/bin/perl
00601000-00603000 rw-p 00001000 08:01 2229974                            /usr/bin/perl
0146c000-3f90ce000 rw-p 00000000 00:00 0                                 [heap]
372b800000-372b820000 r-xp 00000000 08:01 4587522                        /lib64/ld-2.12.so
...(alot of these)....
=>> PBS: job killed: vmem 17400672256 exceeded limit 17179869184

When I ran a script that I have used many times before (smallish files 5-10gb) on a larger file (50gb).  I googled around and don't really understand what my code problem could be in relation to those answers so here's my code:
 my ($ethnicity) = @ARGV;
 my @map;

open ($fh, "<", "/path/$filename")  or die ("Unable to open input file: $!");

open ($fh4, "<", "/path/$filename")     or die ("Unable to open input file: $!");
my @ID = <$fh4>;

open ($fh2, ">", "/path/$filename") or die ("Unable to open output file: $!");
open ($fh3, ">", "/path/$filename") or die ("Unable to open output file: $!");


my $i=0;
my $j=0;

print "The .ped and .map files are being generated\n";

while(  my $line = <$fh> ){
if ($.<3){ 
     $map[$j] = $line;
     $j++;}
else{
    my $temp = $line;
    $temp =~ m/(\d*) (.*)/;
    chomp($temp);
    my $ID = $1;
    print "$ID- ";
    my $geno = $2;
    $geno =~ s /([ATCG])([ATCG])/$1 $2/g;

    $temp = $ID[$i];
    $temp =~ m/(\d*) (1 0 0 [012] [012])/;
    print "$1\n";
    if($ID eq $1){
          print $fh2 "$ID $2 $geno\n"; 
          $i++;}
    else{next;}
    }
}

close $fh;
close $fh2;

What have I done wrong?  I thought I coded it to look line by line, and write out to the file line by line to save memory.  

Any reason to believe "Out of memory!" (when the size of your program exceeded 16GiB) means something other than you ran out of memory?

hmm i really dont think so - the cluster has >300gb of space to use

memory (RAM), not disk space

commands says:   unlimited

Well, something is limiting your process's size to 16GiB, but I don't know what. Not my area.

PBS appears to be the name of a job scheduler. Check its settings.

Yes, it is  but the problem is somewhere in the perl code, not the scheduler.  The code also quits not on a scheduler.

This line (my @ID = <$fh4>;) reads in the entire file pointed to by fh4.  Is this desirable?

You're loading the entire file opened as $fh4 into memory, but there's no reason to.
my $line_from_fh4 = <$fh4>;
my ($id_from_fh4, $rest_from_fh4) = $line_from_fh4 =~ m/(\d*) (1 0 0 [012] [012])/;

while(my $line = <$fh>) {
    if ($.<3) { 
        push @map, $line;
        next;
    }

    chomp($line);

    my ($ID, $geno) = $line =~ m/(\d*) (.*)/;
    $geno =~ s/[ATCG]\K(?=[ATCG])/ /g;

    print "$ID- $id_from_fh4\n";

    if ($ID eq $id_from_fh4){
        print $fh2 "$ID $rest_from_fh4 $geno\n"; 
        $line_from_fh4 = <$fh4>;
        ($id_from_fh4, $rest_from_fh4) = $line_from_fh4 =~ m/(\d*) (1 0 0 [012] [012])/;
    }
}


Wow - this reduced a lot of my code :) but it dosen't work.  I tired $fh4 is still begin read into the whole file with this line:: my $line_from_fh4 = <$fh4>;  also, this dosen't account for when i need it to go the next geno line, but stay on teh $fh4 line until a match does occuer

<$fh4> in scalar context (as it is in my $line_from_fh4 = <$fh4>;) reads one line, where a line is defined as "until the first newline is encountered" by default. So if what you say is true, your file only has one line or you changed the default (by changing $/).

Re "this dosen't account for when i need it to go the next geno line, but stay on teh $fh4 line until a match does occuer", Yes it does. It only goes to the next line of $fh4 when a match occurs.
