Speaking of logs, I want to put some logging in place for my web server. I log every single request with extensive details, so I can debug things later if needed. It&#x27;s several gigabytes per day now, so I can no longer just dump it on disk as I did for the last couple of years.<p>Since I&#x27;m on AWS EC2, I want to try this:<p><pre><code>  - Write the logs to local SSD, asynchronously 
    so as not hold back the http request.
  - Have a separate cron job that loops through 
    the log directory and scoops up all the files.
  - The job will then stuff those files into a Kinesis Firehose.
    AFAIK, Kinesis Firehose does not require any capacity provisioning, 
    unlike the Kinesis Streams, so I&#x27;m set &quot;for life&quot; (up to 5MB&#x2F;second)
  - The firehose will accumulate the logs and put them into S3. 
    Hurray unlimited storage!
  - S3 will trigger a Lambda.
  - Lambda will parse through the log from S3, pull out 
    interesting properties (IP address, user id, session id, etc) and 
    stuff them into a DynamoDb table.
  - If I need to see data from one user&#x2F;ip&#x2F;session I will use DynamoDb 
    to find the right S3 blobs.
  - If I need to reprocess the logs to extract a new piece 
    of data that I did not foresee earlier, I can run a 
    map-reduce task
</code></pre>
Except the last piece, this looks like something I can half-ass in a couple of days and forget about it for another couple of years.<p>Any opinions? I don&#x27;t really want to use a SaaS log service because gigabytes per day.