Question: I am processing large text files (~20MB) containing data delimited by line.
Most data entries are duplicated and I want to remove these duplications to only keep one copy.

Also, to make the problem slightly more complicated, some entries are repeated with an extra bit of info appended. In this case I need to keep the entry containing the extra info and delete the older versions.

e.g.
I need to go from this:

```

BOB 123 1DB
JIM 456 3DB AX
DAVE 789 1DB
BOB 123 1DB
JIM 456 3DB AX
DAVE 789 1DB
BOB 123 1DB EXTRA BITS

```

to this:

```

JIM 456 3DB AX
DAVE 789 1DB
BOB 123 1DB EXTRA BITS

```

NB. the final order doesn't matter.

What is an efficient way to do this?

I can use awk, python or any standard linux command line tool.

Thanks.

Answer: 
`awk '{x[$1 " " $2 " " $3] = $0} END {for (y in x) print x[y]}'`

If you need to specify the number of columns for different files:

```
awk -v ncols=3 '
  {
    key = "";
    for (i=1; i<=ncols; i++) {key = key FS $i}
    if (length($0) > length(x[key])) {x[key] = $0}
  }
  END {for (y in x) print y "\t" x[y]}
'

```
The function `find_unique_lines` will work for a file object or a list of strings.

```
import itertools

def split_line(s):
    parts = s.strip().split(' ')
    return " ".join(parts[:3]), parts[3:], s

def find_unique_lines(f):
    result = {}
    for key, data, line in itertools.imap(split_line, f):
        if data or key not in result:
            result[key] = line
    return result.itervalues()

test = """BOB 123 1DB
JIM 456 3DB AX
DAVE 789 1DB
BOB 123 1DB
JIM 456 3DB AX
DAVE 789 1DB
BOB 123 1DB EXTRA BITS""".split('\n')

for line in find_unique_lines(test):
        print line
```

```
BOB 123 1DB EXTRA BITS
JIM 456 3DB AX
DAVE 789 1DB
```