Scaling Django to 8 Billion Page Views
I applaud Disqus for scaling Django to this tier of sustained load.  I applaud them for sharing a clearly-written and approachable explanation of how that was achieved.  I also applaud them for their product in general.  I think Disqus is a quite excellent embeddable comment tool.<p>I do have some reservations with a few points made by this article.  (Below I am speaking generally, and not about Disqus in particular.  I don't mean anything below to imply they are doing it wrong.  On the contrary, I think they're doing it very right given their circumstances.)<p>Repeated is the conventional wisdom that the performance of your application logic is negligible versus external systems such as your database server or your back-end cache.  For low-performance frameworks and platforms that is indeed commonly the case, hence the conventional wisdom.  However, there are important caveats: first, do not confuse time spent in your database driver and ORM as waiting for your database server.  Your database server vendor will find that hurtful and offensive.  Most database servers will be able to retrieve rows from well-indexed tables at far greater rates than low-performance application platforms' ORMs can translate those rows into usable objects.  Modern database servers fetching rows from well-indexed tables can keep up with the query demands of the very highest-performance frameworks without saturating a database server's CPUs (with throughput measured in the tens to hundreds of thousands of queries per second per server).  Yes, at scale your database server may need attention.  But it's not necessarily the pain point you might think it is.  Bottom line: profile your application and watch your database server's performance metrics.  You may not be waiting on your database despite conventional wisdom.  The same is true for other third-party systems such as a back-end cache.<p>Coupling the above with application logic and in-application composition of content into client-digestable markup ("server side templates") will compound the impact of a low-performance platform.  While high-performance platforms can execute application logic and compose a server-side template tens of thousands of times per second on modest hardware, low-performance platforms may suffer a ten-times or greater performance penalty by comparison.<p>It is not necessarily true that high-performance frameworks and platforms are lower-productivity if you are starting with a green-field scenario where your development team is free of incumbent preferences.  That last bit is crucial, of course.  Most teams do have preferences, past experience that can be leveraged, and "know-how" with legacy frameworks.  Do not confuse this institutional knowledge with an objective measure of developer efficiency.  Developers who are unfamiliar with both Django and a modern high-performance framework may see roughly equal productivity.  Measuring your Django-experienced teams' productivity versus their productivity with (for the sake of argument) a Go framework or a modern JVM framework is a biased assessment because of the alternative's learning curve.  If we continue to judge net productivity as a combination of learning curve <i>and</i> the resulting and ongoing effort level past the learning curve, little with a learning curve will be honestly evaluated.<p>Yes, reverse proxy caching such as that provided by Varnish is an excellent idea when your application is a public-facing system without a great deal of personalization.  But not all systems are public-facing embeddable comments or blogs or news sites (I don't mean this to be critical!).  In many systems, a majority of responses are tailored to the specific user and other entities making them unavailable for caching (as the article mentions these requests will typically use a cookie to identify the session and are therefore not cached by Varnish).  In these cases, if it weren't already clear from the above, I recommend seriously considering a higher-performance platform and framework that gives you the headroom to deliver responses under high load without necessarily resorting to crutches like a reverse proxy.  Yes, leverage caching where-ever and when-ever possible.  But when you cannot cache, respond as quickly as possible.<p>Performance is actually an important concern.  It's not <i>the</i> concern, but don't keep throwing it under the bus.<p>Further, performance is not only a scale and concurrency concern.  It's also a user-experience matter.  In addition to reducing the system complexity for high-load and high-concurrency, a high-performance platform means that even without load and concurrency, you are able to respond to user requests more <i>quickly</i> (reduced latency).  This leads to user happiness, and in some circumstances better search engine positioning and similar fringe benefits.<p>Again, I want to be clear that I think Disqus is great and this article is a valuable contribution, especially for those who are invested in a similar technology stack with similar usage characteristics.