title: Parsing intertwined multiline logs with logstash
body: <p>I am trying to parse logs that are of the following form:</p>
  
  <pre><code>07/12/16 00:41:03.30 FABsrv.1:fwsrvExecServerRule:FwSelectRawSqlTxn stopped at 00:41:03 , and took 0.000413000
  07/12/16 00:41:03.30 FABsrv.1:fwsrvExecServerRule:FwSelectRawSqlTxn started at 00:41:03
  07/12/16 00:41:03.30 FABsrv.1:fwsrvExecServerRule:FwSelectRawSqlTxn stopped at 00:41:03 , and took 0.000450000
  07/12/16 00:41:03.30 FABsrv.15:FwEventManager:message to process: :&gt;&gt;L FwSrvExecuteRuleTxn msgTag=_INBOX.0A121933.DC05784206C6061BD8.13 ruleName=C2_BATCHCHECK attributes={ { class=ASSOC { class=A1 "PARAMETER" } { class=A1 "VALUE" } } { "USERID" "BC212" } { "EQPID" "FAITU02" } { "LOTIDS" "Q619294" } { "LOTSPPIDS" "Q619294=B401HA/DEFFIN_A" } { "FORCENOMEASURE" "F" } { "FORCEBATCHMAX" "F" } { "TID" "CRO06666_RN004102950_V5.8.0.0004" } }
  07/12/16 00:41:03.30 FABsrv.15:fwsrvExecServerRule:FwSelectRawSqlTxn started at 00:41:03
  07/12/16 00:41:03.30 FABsrv.1:fwsrvExecServerRule:FwSelectRawSqlTxn started at 00:41:03
  07/12/16 00:41:03.30 FABsrv.1:fwsrvExecServerRule:FwSelectRawSqlTxn stopped at 00:41:03 , and took 0.001052000
  07/12/16 00:41:03.30 FABsrv.15:fwsrvExecServerRule:FwSelectRawSqlTxn stopped at 00:41:03 , and took 0.001576000
  07/12/16 00:41:03.30 FABsrv.15:fwsrvExecServerRule:FwSelectRawSqlTxn started at 00:41:03
  07/12/16 00:41:03.31 FABsrv.15:fwsrvExecServerRule:FwSelectRawSqlTxn stopped at 00:41:03 , and took 0.000910000
  07/12/16 00:41:03.31 FABsrv.15:C2_BATCHCHECK:CAPITraceDiagnostic - finish executing rule C2_BATCHCHECK:Reply
  </code></pre>
  
  <p>They are generated by a pool of threads running on some server that all write to a single file and I need to extract information from blocks of them keeping in mind the following constraints:</p>
  
  <ul>
  <li><p>Events are multiline, those <code>stopped</code> and <code>started</code> you see can have lines between them and those need to be joined together</p></li>
  <li><p>Events are intertwined, the third column specifies which thread wrote which line and each thread needs to correspond to a different stream of data</p></li>
  <li><p>I need a thread safe solution</p></li>
  </ul>
  
  <p>That last point is the real problem, since what is currently in use is the <code>multiline</code> filter with its <code>stream-identity</code> option to split the stream based on the thread number, but that solution can't scale to the amount of data we have now and the <code>mutiline</code> filter is deprecated in favor of<code>multiline-codec</code> which is thread safe, but does not support that stream-splitting functionality.</p>
  
  <p>Splitting the logs into mutiple files would be my obvious solution but this is not possible for operational reasons.</p>
  
  <p>Is there any way to do this?</p>