Django: Simple rate limiting
What about using a different process to handle scraping, and a queue for the communication between it and Django?
This way you would be able to easily change the number of concurrent requests, and it would also automatically keep track of the requests, without blocking the caller.
Most of all, I think it would help lowering the complexity of the main application (in Django).
Many of my views fetch external resources. I want to make sure that under heavy load I don't blow up the remote sites (and/or get banned).
I only have 1 crawler so having a central lock will work fine.
So the details: I want to allow at most 3 queries to a host per second, and have the rest block for a maximum of 15 seconds. How could I do this (easily)?
Some thoughts :

Use django cache


Seems to only have 1 second resolution

Use a file based semaphore


Easy to do locks for concurrency. Not sure how to make sure only 3 fetches happen a second.

Use some shared memory state


I'd rather not install more things, but will if I have to.