Question: I have around a 50GB folder full of files. Each file consists of line after line of JSON data and in this JSON structure is a field for `user_id`.

I need to count the number of unique User IDs across all of the files (and only need the total count). What is the most memory efficient and relatively quick way of counting these?

Of course, loading everything into a huge list maybe isn't the best option. I tried pandas but it took quite a while. I then tried to simple write the IDs to text files but I thought I'd find out if I was maybe missing something far simpler.

Answer: 
Try the simplest approach first.

Write a function `get_user_ids(filepath)` that returns a list of `user_id` in a JSON file.

Then do:

```
from pathlib import Path
the_folder = Path("path/to/the/folder")
user_ids = set()
for jsonpath in the_folder.glob('*.json'):
    user_ids.update(get_user_ids(jsonpath))
print(len(user_ids))

```
If the list of user IDs is so large that it can't reasonably fit into a set in memory, an easy and memory-efficient way to de-duplicate is to simply create files named after user IDs in an empty directory, and then count the number of files in the directory. This works because most filesystems are efficient at indexing file names in a directory.

```
import os
os.chdir('/')
os.mkdir('/count_unique')
os.chdir('/count_unique')
# change the following demo tuple to a generator that reads your JSON files and yields user IDs
for user_id in 'b', 'c', 'b', 'a', 'c':
    open(user_id, 'w').close()
print(sum(1 for _ in os.scandir('/count_unique')))

```

This outputs: `3`