This was a fun Saturday afternoon project to do actually. I spawned a single c4.8xlarge (10 Gbit) instance in US-EAST-1 (next to where the Common Crawl Public Data Set lives in S3) and downloaded 10TB spread over 33k files in +&#x2F;- 30 simultaneous &#x27;curl | ungzip | grep whatever-the-common-crawl-url-prefix-was&#x27; pipelines and got a solid 5Gbit transfer speed.<p>The bottleneck was userland CPU, so probably the gzip processes. Took about 5 hours. Cutting of the path and piping through sort and uniq took another hour or so.<p>The only costs were the price&#x2F;hour of running the EC2 instance. Network costs were 0 as you&#x27;re only transferring data to your instance.