I have a huge database of over 20 million rows. I can export the whole database (which takes hours), but when I try to filter the data using python (pandas) pycharm fails (due to memory issues).
Is there a way to export the database in batches of 2 million rows for an example? Export 2mil, then other 2mil and have 10 files of 2 million rows at the end? This way I can filter every file using python (pandas) and I won't have memory issues.
Thanks!
You can use pg_dump to only extract one or more tables or exclude tables if that is going to help
How to export huge postgresql database in batches?