Celery, non-blocking code and quest against coroutines
There is opinionated, and then there is stupidly opinionated.  This article crosses that line then stays there.  It offers one useful point, but that one is hardly news.<p>For example we have claims like this, <i>Do people use coroutines? Yes, but not in production code.</i>  This claim is trivially false.  There is lots of production code that has the word &quot;yield&quot; in it.  Heck, plenty of production code uses Tornado, which at its heart is nothing BUT coroutines.  And most programmers have no trouble reading it.<p>Moving on we see, <i>Also, cooperative concurrency aka coroutines was abandoned a fair amount of time ago. I wonder why?</i>  I&#x27;m sure that people who use Redis and Node.js would be surprised to discover that cooperative concurrency has no place in our modern software world.<p>He even concludes that people don&#x27;t need concurrency.  Which is obviously ridiculous - just look at any distributed system.<p>But let&#x27;s go back to why people move from cooperative concurrency to preemptive.  Answer it will make it clear why we&#x27;ll always have cooperative concurrency in the programming mix.<p>Cooperative concurrency means that you don&#x27;t need locks and don&#x27;t have race conditions.  However one poorly programmed routine can block your whole system.  Preemptive concurrency means that a bad routine can no longer block your whole system, but now you have opened up the guarantee of complex race conditions.  Programmers are really, really bad at understanding race conditions.  So bad that almost all of our software has them.  So bad that people who write automated code verification tools have learned the hard way to not report on them because programmers won&#x27;t understand what you are talking about.  (Sadly true.  In <a href="http:&#x2F;&#x2F;web.stanford.edu&#x2F;~engler&#x2F;BLOC-coverity.pdf" rel="nofollow">http:&#x2F;&#x2F;web.stanford.edu&#x2F;~engler&#x2F;BLOC-coverity.pdf</a> look for the phrase, &quot;...for many years we gave up on checkers that flagged concurrency  errors.&quot;)<p>Therefore if you want concurrency with no locking overhead, and no race conditions, then coroutines are great.  But once your application becomes big and messy enough, it becomes better to accept the problems that come with preemption.<p>Now if you&#x27;re going to preempt, there is a ton of advice to be offered on how to do it well.  For example you&#x27;ll generally have fewer problems with coarse-grained processes than with threads.  Queues and worker pools are a great solution for some types of problem, and I first did that about 15 years ago.  It was hardly an original idea when I first did it - it was suggested to me by someone who had been using that technique since the 1980s.  And it wasn&#x27;t original with him either.<p>However there are a <i>ton</i> of use cases that this doesn&#x27;t cover.  For example it doesn&#x27;t do much to help you handle high volumes of traffic to your webserver, and it won&#x27;t let you create a distributed data store for low-latency access to terabytes of data.  So put celery in your tool bag, know what it is good for, and don&#x27;t make the mistake of thinking that it solves all problems.