So, I have a local office that sends us a list of new subscribers from their location daily. But instead of just sending over the new data, they just send over a csv with what i think is the last 25,000 records - i'm guessing the clerk just clicks some default option.
I have a simple python script which inserts this data to a local mysql db, with sub_id set as unique index to prevent duplicates.
My problem however, is that I have to send the new subscriber data over to another team.
I want to add this functionality to the existing python script, and the solution I could think of is to add a "NEW" status when inserting to the db, and then export all rows with "NEW" status, and then update the "NEW" status to "EXPORTED".
This feels inefficient to me -
Is there a better approach to this?
Add a created_at timestamp to the table and send everything with a created_at greater than when the process started. This is the most robust option, and also adds a bit of information to the database which may be useful later; you can figure out which row was made by which import.
Or, have the Python script remember the IDs of every row it inserted. Then select only those rows.
Alternatively, have the Python script build the report as it inserts the data.
Insert only unique data to db and export new data to csv