Answers: <p>One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.</p>
  
  <p>To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, <a href="https://www.elastic.co/start" rel="noreferrer">https://www.elastic.co/start</a>, or it's also possible to run it in Docker. The <code>stack-docker</code> repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability. </p>
  
  <p>The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.</p>
  
  <p>Some more best practices:</p>
  
  <ul>
  <li>Log all your messages in JSON, with a focus on objects/fields rather than long string messages.</li>
  <li>Exceptions should be actionable, anything else can be a warning.</li>
  <li>Avoid multiline logs.</li>
  <li>Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.</li>
  <li>Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.</li>
  <li>Separate your environments logically so you can have different retention periods and archiving on your logs.</li>
  </ul>
  
Answers: CC BY-SA 3.0
Answers: 2017-10-09T08:20:12.317Z
Answers: 2017-10-09T08:43:27.903Z
Answers: 2017-10-09T08:43:27.903Z
Answers: user2640621
Body: <p>At the moment there are multiple microservices that log to individual files. If there is an issue then the team is debugging like headless chickens. In order to prevent this the aim is to introduce aggregated logging, e.g. <a href="https://www.elastic.co/" rel="nofollow noreferrer">elastic</a>. What are best practices to migrate to such an aggregated logging system?</p>
  
  <p><strong>Current idea</strong></p>
  
  <ul>
  <li>Deploy <a href="https://github.com/elastic/stack-docker" rel="nofollow noreferrer">an elastic stack locally</a></li>
  <li>navigate to <a href="http://localhost:5601" rel="nofollow noreferrer">http://localhost:5601</a></li>
  <li>login by entering <code>elastic</code> and <code>changeme</code></li>
  <li>get the endpoint and configure it in every microservice? (<a href="https://www.elastic.co/guide/en/cloud/current/getting-started.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/cloud/current/getting-started.html</a>)</li>
  </ul>
  
  <p>What endpoint should be configured?</p>
  
Comments: CC BY-SA 3.0
Comments: 2017-10-09T05:32:49.257Z
Comments: Is this a "cultural" or a technical question? Did you try what you wrote, did anything go wrong?
Comments: AnoE
Tags: <elastic>
Title: What are best practices for implementing and using Elastic in an organization?
UserDisplayName: 030
ThreadText: # What are best practices for implementing and using Elastic in an organization?
  
  (Asked by: username_0 on 2017-10-08)
  
  At the moment there are multiple microservices that log to individual files. If there is an issue then the team is debugging like headless chickens. In order to prevent this the aim is to introduce aggregated logging, e.g. [elastic](https://www.elastic.co/). What are best practices to migrate to such an aggregated logging system?
  
  **Current idea**
  
  * Deploy [an elastic stack locally](https://github.com/elastic/stack-docker)
  * navigate to <http://localhost:5601>
  * login by entering `elastic` and `changeme`
  * get the endpoint and configure it in every microservice? (<https://www.elastic.co/guide/en/cloud/current/getting-started.html>)
  
  What endpoint should be configured?
  
  **Tags:** elastic
  
  > **Comments:**
  > **username_1**: Is this a "cultural" or a technical question? Did you try what you wrote, did anything go wrong?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-10-09. Score: 5 | Accepted answer
  
  One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.
  
  To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, <https://www.elastic.co/start>, or it's also possible to run it in Docker. The `stack-docker` repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability.
  
  The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.
  
  Some more best practices:
  
  * Log all your messages in JSON, with a focus on objects/fields rather than long string messages.
  * Exceptions should be actionable, anything else can be a warning.
  * Avoid multiline logs.
  * Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.
  * Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.
  * Separate your environments logically so you can have different retention periods and archiving on your logs.
  
  
  ---
  
  