Asynchronous Python and Databases
If you have a Python CRUD app, there are many steps you should take to speed up the app before parallelizing queries.<p>The first couple that come to mind:<p>- Caching. The 90-10 rule applies to most CRUD apps. Use varnish to get rid of some requests before they hit your webserver or make a redis/memcache LRU layer to prevent queries from hitting your database.<p>- Shifting more work away from Python to the database. A surprising number of apps don't do pagination properly.<p>For example, if you want to show the top 10 X objects for each item Y, you don't need to get X * Y rows. You can use a window function to get the top 10 for each Y can get 10 * Y rows instead.<p>- Using PyPy if you can. It's a free performance boost, in most cases.<p>- Smarter indexing. Postgres's partial indexes are really powerful.<p>- Intermediate caches. If you have common GROUP BY queries, a materialized view could go a long way.<p>- If you reallllllly need it, you can Cython and rewrite slow parts of your app in C.<p>My point is - parallelizing queries should be one of your last steps in speeding up your app. It adds a lot of complexity sometimes and there's a lot of cheap easy optimizations out there.