Python big list and input to database
If these many links are spread across several files, what about a commit after processing each file? Then you as well could remember which files you have processed.
In the case of a single file, record the file offset after each commit for clean continuation.
I must parse HTML files, which can be up to 500 000 links.
Of which 400 000 will be desired by me.
Should I put all the links that satisfy the condition for the new list and then for the elements of this list and put it into the database.
Or when I find links to satisfy the condition to add it to the database (sqlite) (and commit it).
Is that a large number of commits is not a problem?
I do not want to lose data in case of failure such as power. Thats why i want commit after  insert to the database.
How best to place a large number of items in the database?