Asynchronous Python and Databases
If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries.The first couple that come to mind:- Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database.- Shifting more work away from Python to the database. A surprising number of apps don't do pagination properly.For example, if you want to show the top 10 X objects for each item Y, you don't need to get X * Y rows. You can use a window function to get the top 10 for each Y can get 10 * Y rows instead.- Using PyPy if you can. It's a free performance boost, in most cases.- Smarter indexing. Postgres's partial indexes are really powerful.- Intermediate caches. If you have common GROUP BY queries, a materialized view could go a long way.- If you reallllllly need it, you can Cython and rewrite slow parts of your app in C.My point is - parallelizing queries should be one of your last steps in speeding up your app. It adds a lot of complexity sometimes and there's a lot of cheap easy optimizations out there.