What's the Fastest way to get a large number of files (relatively small 10-50kB) from Amazon S3 from Python? (In the order of 200,000 - million files). 
At the moment I am using boto to generate Signed URLs, and using PyCURL to get the files one by one. 
Would some type of concurrency help? PyCurl.CurlMulti object? 
I am open to all suggestions. Thanks!
I've been using txaws with twisted for S3 work, though what you'd probably want is just to get the authenticated URL and use twisted.web.client.DownloadPage (by default will happily go from stream to file without much interaction).
Twisted makes it easy to run at whatever concurrency you want.  For something on the order of 200,000, I'd probably make a generator and use a cooperator to set my concurrency and just let the generator generate every required download request.
If you're not familiar with twisted, you'll find the model takes a bit of time to get used to, but it's oh so worth it.  In this case, I'd expect it to take minimal CPU and memory overhead, but you'd have to worry about file descriptors.  It's quite easy to mix in perspective broker and farm the work out to multiple machines should you find yourself needing more file descriptors or if you have multiple connections over which you'd like it to pull down.
Downloading a Large Number of Files from S3