Python Batch algorithm for getting consecutive events(streaks) from app user data
One of my recent project required to create a job that pulls in users activities with a timestamp and build the longest and the latest streaks for all users in a batch.

I ended up writing a job in python and hooked it up to an hourly worker. The job mainly does these four things in a series

1. pull in (userId, createdAt as t) records from a source table

2. load them into a dataframe and group consecutive days for each user

3. get the max and latest consecutive days for each user

4. write streaks in batch to a target table

Basically, the task pulls in these source records

and builds out this target records

Implementation is available here, I have also shared a notebook file for showing details for step #2 and #3 here

Method buid_streak_counts in the batch_streak_count.py is the task executed by an hourly job, and at the end, it writes streaks into a target table via Postgres specific batch write operation copy_from which I found ridiculously fast.

It has been in running in production for the last few weeks and has been quite robust so far.

Please feel free to comment on anything!
['Seoyeon Chang']
['Postgres', 'Data Science', 'Streaks', 'Python', 'Pandas']