Move Fast and Migrate Things: How We Automated Migrations in Postgres
I am convinced that data migration is definitely one of the hardest problems in data management and systems engineering.<p>There are basically no solutions today that satisfy fundamental requirements such as minimizing downtime and guaranteeing correctness.<p>It is _such_ a huge problem that most inexperienced developers see kicking the problem down the line with NoSQL document storage as a viable alternative (it isn&#x27;t; you&#x27;ll be either dealing with migrating all data forever and special-casing every old version of your documents, or writing even more convoluted migration logic).<p>It&#x27;s also clear that even the most modern ORMs and query builders have not been built in mind to consider the issues that arise in migrating data.<p>It would be a refreshing thing to see more research devoted to this problem. Unfortunately, migrations end up being so different from each other with such heterogenous requirements that we&#x27;ll probably be working on this for a really long time.