Hi all,

I've written one perl script, which compares two vcf files and write output in third file. it is running good with small files, but when i am inputting big files in Gbs, my system becomes slow and it would hang.

I'm using dbSNP vcf file as input which is around 9GB.

Please Suggest me something that will make my perl script run faster. This is my first perl script and i'm new to perl.

Any help would be appreciated !!

Thank you !!!!

```
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use List::MoreUtils;


open (FILE, "<", "dbSNP_in.vcf") or die "failed to open: $!\n";
my @array=(<FILE>);
my @CHR;
my @location;
my @rs;
my @ref_n;
my @alt_n;
foreach (@array)
{
    chomp;
    my ($chrom, $pos, $id, $ref, $alt, $qual, $filter, $info) = split(/\t/, $_);

push @CHR, $chrom;
push @location, $pos;
push @rs, $id;
push @ref_n, $ref;
push @alt_n, $alt;
}

open (FILE1, "<", "trial_rs.vcf") or die "failed to open: $!\n";
my @array1=(<FILE1>);

open (OUT,">trial_output.vcf");
my @columns;

foreach (@array1)
{
    chomp;
    @columns=split(/\t/, $_);
    my $i;
    for ($i=0; $i<@array; $i++)
    {
        if (($columns[0] eq $CHR[$i]) and ($columns[1] eq $location[$i]) and ($columns[3] eq $ref_n[$i]) and ($columns[4] eq $alt_n[$i]))
        {
            $columns[2]=$rs[$i];
        }

    }

print OUT join("\t", @columns), "\n";
}
```
The problem is this:

    my @array=(<FILE>); ## slurp a whole Terabyte into RAM

Do not read a file like this, unless it is very small or your memory is huge, instead use:

```
while (<FILE>) { # read each line, then forget about it
    chomp;
    split /t/;
    ....
}
```

In addition your code is extremely inefficient:

You are iterating over all entries of a huge array for each line of a huge file. Hashes in Perl provide constant time access to elements given the hash key. Use a hash of hashes to store your filter table.

You seem to filter a large file by a small file, therefore:

 - process the small file first (using while construct), parse the small file into a hash using the location as key of a nested hash
 - process the large file as above and look up each of the entries in the hash created before

As a result you will only need as much memory as is required to store the small file.