Answers: <p>If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:</p>
  
  <ul>
  <li>basic context (timestamp, log level, application name, source file name, source module/function, source line number, log message)</li>
  <li>server context (host name, data center name unless host name has it encoded, cluster/pod info, container info)</li>
  <li>user context (request URI, username or a hash, request ID etc)</li>
  </ul>
  
  <p>If we are using JSON for logging, care must be taken to make sure that the keys are short enough to reduce the redundant overhead and also make it easier to query.  Using all lowercase and underscores for keys could be a better idea because some of the log aggregators like Splunk are case sensitive.</p>
  
  <p>In the structured format, stack traces become a little tricky to handle.  One solution is to use a marker line to indicate the end of a long line rather than just the newline.</p>
  
Answers: CC BY-SA 4.0
Answers: 2019-07-31T05:29:55.420Z
Answers: Thanks very much - I did indeed neglect to mention that we are talking about application logs here, as opposed to http or server logs.
Answers: Bruce Becker
Answers: CC BY-SA 4.0
Answers: 2019-07-30T20:57:50.563Z
Answers: 2019-07-31T05:48:05.980Z
Answers: 2019-07-31T05:48:05.980Z
Answers: codeforester
Body: <p>We have a practical question I struggle to answer : </p>
  
  <blockquote>
    <p>What is a good logfile schema?</p>
  </blockquote>
  
  <p>Here are the caveats:</p>
  
  <ul>
  <li>There are several workloads, each writing more than one log file</li>
  <li>Applications are on more than one language</li>
  <li>All applications are in the cloud, but on different cloud providers</li>
  <li>All logs are sent to a central dashboard (e.g. Kibana) which can be queried.</li>
  </ul>
  
  <p>The requirements which this schema should fulfill are:</p>
  
  <ul>
  <li>must be queryable (structured)</li>
  <li>should be able to determine failure patterns (correlations)</li>
  <li>no proprietary file formats</li>
  </ul>
  
  <p>For a particular case, we may want to set a trigger of some action based on a particular query.</p>
  
  <p>Assuming, e.g. we write logs in JSON format, what would be a good schema to apply to the entries, to satisfy the caveats and requirements?</p>
  
Tags: <monitoring><cloud><logging><methodology>
Title: What considerations should be taken into account for a good log file format?
UserDisplayName: Bruce Becker
ThreadText: # What considerations should be taken into account for a good log file format?
  
  (Asked by: username_0 on 2019-07-30)
  
  We have a practical question I struggle to answer :
  
  > What is a good logfile schema?
  
  Here are the caveats:
  
  * There are several workloads, each writing more than one log file
  * Applications are on more than one language
  * All applications are in the cloud, but on different cloud providers
  * All logs are sent to a central dashboard (e.g. Kibana) which can be queried.
  
  The requirements which this schema should fulfill are:
  
  * must be queryable (structured)
  * should be able to determine failure patterns (correlations)
  * no proprietary file formats
  
  For a particular case, we may want to set a trigger of some action based on a particular query.
  
  Assuming, e.g. we write logs in JSON format, what would be a good schema to apply to the entries, to satisfy the caveats and requirements?
  
  **Tags:** monitoring, cloud, logging, methodology
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-30. Score: 2 | Accepted answer
  
  If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:
  
  * basic context (timestamp, log level, application name, source file name, source module/function, source line number, log message)
  * server context (host name, data center name unless host name has it encoded, cluster/pod info, container info)
  * user context (request URI, username or a hash, request ID etc)
  
  If we are using JSON for logging, care must be taken to make sure that the keys are short enough to reduce the redundant overhead and also make it easier to query. Using all lowercase and underscores for keys could be a better idea because some of the log aggregators like Splunk are case sensitive.
  
  In the structured format, stack traces become a little tricky to handle. One solution is to use a marker line to indicate the end of a long line rather than just the newline.
  
  > **Comments:**
  > **username_0**: Thanks very much - I did indeed neglect to mention that we are talking about application logs here, as opposed to http or server logs.  
  
  ---
  
  