Question: What i'm trying to do is to read big file 5.6GB have approximately 600Million lines and the second is 16MB have 2M lines.

I want to check the duplicate lines in these two files.

```
$wordlist = array_unique(array_filter(file('small.txt', FILE_IGNORE_NEW_LINES)));
$duplicate = array();
if($file = fopen('big.txt', 'r')){
    while(!feof($file)){
        $lines = rtrim(fgets($file));
        if(in_array($lines, $wordlist)){
            echo $lines." : exists.\n";
        }
    }
    fclose($file);
}

```

But this take forever to finish ( its been running from 6 hours and didn't finish yet :/ ).

My question is. Is there a better way to search in huge files fast?

Answer: 
You won't need to call `array_filter()` or `array_unique()` if you are going to call `array_flip()` -- it will eliminate the duplicates for you because you can't have duplicate keys in the same level of an array.

Furthermore:

1. `array_unique()` is stated to be slower than `array_flip()` (and there are times when it is slower than two `array_flip()`s)
2. `array_filter()` has a bad reputation for killing falsey/empty/null/zero-ish data, so I will caution you not to use its default behavior.
3. `array_flip()` sets up the very speedy `isset()` check. `isset()` will likely outperform `array_key_exists()` because `isset()` doesn't check for `null` values.
4. I am adding the `FILE_SKIP_EMPTY_LINES` flag to `file()` call so that your lookup array is potentially smaller.
5. Calling `rtrim()` of every line of your big file, may be causing some drag too. Do you know if you have consistently identical newline characters on both files? It would spare you six hundred millions calls of `rtrim()` if you can safely remove the `FILE_IGNORE_NEW_LINES` flag from the `file()` call. Alternatively, if you *know* the newlines (e.g. `\n`? or `\r\n`?) that trail the big.txt lines, you can append specific newline(s) to the `$lookup` keys -- this means preparing the smaller file's data versus every line of the big file.

Untested Code:

```
$lookup = array_flip(file('small.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES));
if($file = fopen('big.txt', 'r')){
    while(!feof($file)){
        $line = rtrim(fgets($file));
        if (isset($lookup[$line])) {
            echo "$lines : exists.\n";
        }
    }
    fclose($file);
}

```
i think

```
 $wordlist=array_flip(array_unique(array_filter(file('small.txt', FILE_IGNORE_NEW_LINES))));

```

which you actually use in your code,slows it.It may be better to build the wordslist once and foverer yourself:

```
if($file1 = fopen('big.txt', 'r')){
    if($file = fopen('small.txt', 'r')){
        while(!feof($file)){
            $line=trim(fgets($file));
            if(!isset($wordlist[$line])&&!ctype_space($line)&&!empty($line)){
                $wordlist[$line]=0;
            }
        }
        fclose($file); 
    }
    while(!feof($file1)){
        $line1 = trim(fgets($file1));
        if(isset($wordlist[$line1]))
            $wordlist[$line1]++;            
    }
  fclose($file1); 
}

```

At this step the variable $wordlist contain the list of all lines in your small.txt file and the number of occurences of each line in your big.txt file .
You could use the array like that or filter it to remove null lines.You could also sort the array with uasort to know more about which lines occurs the most and which one occurs less and and you can even go further in your analyze...