Ask HN: How do you manage logs for your backend services?
If you haven&#x27;t read the chapter of 12factor on logging I highly recommend it <a href="https:&#x2F;&#x2F;www.12factor.net&#x2F;logs" rel="nofollow">https:&#x2F;&#x2F;www.12factor.net&#x2F;logs</a><p>This is coming from an ops person, do that and I&#x27;ll be happy. Essentially the goal is to externalize all your log routing to stdout, then wrap tooling around your application to route it wherever you want it to go. It&#x27;s geared toward heroku but same rules apply in docker land and more traditional VM environments.<p>We either send logs ECS -&gt; Cloudwatch for our AWS stuff or docker swarm -&gt; fluentd -&gt; s3 for an onsite appliance (also anything syslog ends up in Sumologic through a collector we host in the environment). From there the logs get consumed by our SIEM (in our case Sumologic, which is a great service). We keep logs hot for 90 days and ship archives back to s3 for a year. Set up the lifecycle management stuff, keeping log files forever is not only a waste but can actively hurt you if they ever get exposed in a breach.<p>I highly recommend formatting your logs JSON (even if you have to do it by hand like in Apache). If you do that and go to Splunk, Sumologic, or ELK all your fields will be populated either automatically or with a single filter. Saves writing or buying your own and if you add a field there&#x27;s no action for you to take.<p>nginx&#x2F;apache default logging is complete trash. Look at the variables they expose, there&#x27;s a lot of stuff in there you&#x27;ll want to add to the log format to make your life loads easier. I have a format I use I&#x27;d be willing to send you if you want it.<p>I don&#x27;t recommend logstash (the L in ELK), ever (except maybe if you&#x27;re java across the board). It&#x27;s way too damn heavy to run on a workload host, fluentd is much lighter (and not java, why would I deploy java for a system tool ever?). Maybe as a network collector you throw syslog at but that would be it.<p>For your use case Sumologics free service would be great. You can get I think up to 200m a day with a weeks retention for free and you&#x27;ll get exposed to what an SIEM can do for you (ingestion rate and retention period are typically how hosted solutions are billed, you&#x27;ll need email with any non-free email domain to get a free account from them). IMO you have to get to some fairly insane log rates for me to ever recommend running ELK stack yourself, it has way too much care and feeding if you want to run it correctly with good security.