title: How to pull specific data from log using logstash and Grok and index as fields in Kibana for every record?
body: <p>Goal is: for each line in the log, there should be a document in elastic containing the 'message' (text after time stamp). Each document should also contain fields for the the project name, plan name, and build #. <strong>&lt;--this is where I'm getting stuck</strong></p>
  
  <p>example log structure in the beginning (atlassian bamboo build logs):</p>
  
  <pre><code>simple 01-Jan-2016 14:26:01  Build TestProj - Framework Code - Build #25 (TST-FC-25) started building on agent .NET Core 2
  simple 01-Jan-2016 14:26:01  .NET-related builds, tests and publishing.
  </code></pre>
  
  <p>I have a Grok to get and create the fields I want - build name, build number, and project name (and have them as fields in Kibana):</p>
  
  <pre><code>%{NOTSPACE:log_entrytype}%{SPACE}(?&lt;timestamp&gt;(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])-\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b-(?&gt;\d\d){1,2}\s*(?!&lt;[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))%{SPACE}Build%{SPACE}%{DATA:BamProjName}%{SPACE}-%{SPACE}%{DATA:BamBuildName}%{SPACE}-%{SPACE}Build%{SPACE}#%{NUMBER:BamBuildNum}
  </code></pre>
  
  <p>However I need these fields available in every record/entry in Kibana. With this other Grok, I can extract the other lines of the log into a log_message field:</p>
  
  <pre><code>grok {   [
             "message", "%{NOTSPACE:log_entrytype}%{SPACE}(?&lt;timestamp&gt;(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])-\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b-(?&gt;\d\d){1,2}\s*(?!&lt;[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]))%{SPACE}%{GREEDYDATA:log_message}"
           ]
       }
  </code></pre>
  
  <p>So do I need to somehow combine these two pattern matches into one, using the 'optional': ()? syntax as described here?: <a href="https://stackoverflow.com/questions/30083719/logstash-optional-fields-in-logfile">link</a></p>
  
  <p>Is my end goal achievable with logstash and the Grok plugin alone? Can I handle this with some type of variable construct within logstash? add_field?</p>
  
  <p>***NOTE: using filebeat for shipping logs, and <a href="https://www.elastic.co/guide/en/logstash/current/multiline.html" rel="nofollow noreferrer">elastic does not recommend the multiline codec </a>, so I'm curious what my other options are</p>