Python big list and input to database
You can try to use noSQL database like mongo. With mongo I add 500.000 documents with 6 fields  in each added about 15 seconds (on my old laptop), and about 0.023 sec on not difficult queries.
I must parse HTML files, which can be up to 500 000 links.
Of which 400 000 will be desired by me.
Should I put all the links that satisfy the condition for the new list and then for the elements of this list and put it into the database.
Or when I find links to satisfy the condition to add it to the database (sqlite) (and commit it).
Is that a large number of commits is not a problem?
I do not want to lose data in case of failure such as power. Thats why i want commit after  insert to the database.
How best to place a large number of items in the database?