Here&#x27;s what I&#x27;ve done for ~200GB&#x2F;day. Let&#x27;s pretend you have server logs with urls that tell you referrer and whether or not the visit action was an impression or a conversion and you want stats by &quot;date&quot;, &quot;referrer domain&quot;, &quot;action&quot;:<p>* Logs are written to S3 (either ELB does this automatically, or you put them there)<p>* S3 can put a message into an SQS queue when a log file is added<p>* A &quot;worker&quot; (written in language of your choice running on EC2 or Lambda) pops the message off the queue, downloads the log, and &quot;reduces&quot; it into grouped counts. In this case a large log file would be &quot;reduced&quot; to lines where each line is [date, referrer domain, action, count] (e.g. [[&#x27;2016-02-24&#x27;, &#x27;news.ycombinator.com&#x27;, &#x27;impression&#x27;, 500], [&#x27;2016-02-24&#x27;, &#x27;news.ycombinator.com&#x27;, &#x27;conversion&#x27;, 20], ...]<p>* The reduction can either be persisted in a db that can handle further analysis or you reduce further first.