title: Parsing a log file with ANTLR
body: <p>I need to parse a Weblogic log file with ANTLR. Here is the example:</p>
  
  <pre><code>Tue Aug 28 09:39:09 MSD 2012 [test] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] Alert - There is no user password credential mapper provider configured in your security realm. Oracle Service Bus service account management will be disabled. Configure a user password credential mapper provider if you need OSB service account support.
  
  Sun Sep 02 23:13:00 MSD 2012 [test] [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] Warning - Timer (Checkpoint) has been triggered with a tick (205 873) that is less than or equal to the last tick that was received (205 873). This could happen in a cluster due to clock synchronization with the timer authority. The current trigger will be ignored, and operation will be skipped.
  Mon Sep 03 10:35:54 MSD 2012 [test] [[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'] Info - 
   [OSB Tracing] Inbound request was received. 
  
   Service Ref = Some/URL
   URI = Another/URL
   Message ID = u-u-i-d
   Request metadata =
      &lt;xml-fragment&gt;
        &lt;tran:headers xsi:type="http:HttpRequestHeaders" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
          &lt;http:Accept-Encoding&gt;gzip, deflate,gzip, deflate&lt;/http:Accept-Encoding&gt;
          &lt;http:Connection&gt;Keep-Alive&lt;/http:Connection&gt;
          &lt;http:Content-Length&gt;666&lt;/http:Content-Length&gt;
          &lt;http:Content-Type&gt;text/xml; charset=utf-8&lt;/http:Content-Type&gt;
          &lt;http:Host&gt;some.host.name&lt;/http:Host&gt;
          &lt;http:SOAPAction&gt;""&lt;/http:SOAPAction&gt;
        &lt;/tran:headers&gt;
        &lt;tran:encoding xmlns:tran="http://www.bea.com/wli/sb/transports"&gt;utf-8&lt;/tran:encoding&gt;
        &lt;http:client-host xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;1.2.3.4&lt;/http:client-host&gt;
        &lt;http:client-address xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;1.2.3.4&lt;/http:client-address&gt;
        &lt;http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;POST&lt;/http:http-method&gt;
      &lt;/xml-fragment&gt;
   Payload =  
  &lt;s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"&gt;&lt;XMLHere/&gt;&lt;/s:Envelope&gt;
  </code></pre>
  
  <p>I am interested in this part of a log, everything else must be ignored (Date, Service Ref value and Envelope XML should be parsed):</p>
  
  <pre><code>Sun Sep 02 23:13:00 MSD 2012 [test] [[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] Warning - Timer (Checkpoint) has been triggered with a tick (205 873) that is less than or equal to the last tick that was received (205 873). This could happen in a cluster due to clock synchronization with the timer authority. The current trigger will be ignored, and operation will be skipped.
      Mon Sep 03 10:35:54 MSD 2012 [test] [[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'] Info - 
       [OSB Tracing] Inbound request was received. 
  
       Service Ref = Some/URL
       URI = Another/URL
       Message ID = u-u-i-d
       Request metadata =
          &lt;xml-fragment&gt;
            &lt;tran:headers xsi:type="http:HttpRequestHeaders" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"&gt;
              &lt;http:Accept-Encoding&gt;gzip, deflate,gzip, deflate&lt;/http:Accept-Encoding&gt;
              &lt;http:Connection&gt;Keep-Alive&lt;/http:Connection&gt;
              &lt;http:Content-Length&gt;666&lt;/http:Content-Length&gt;
              &lt;http:Content-Type&gt;text/xml; charset=utf-8&lt;/http:Content-Type&gt;
              &lt;http:Host&gt;some.host.name&lt;/http:Host&gt;
              &lt;http:SOAPAction&gt;""&lt;/http:SOAPAction&gt;
            &lt;/tran:headers&gt;
            &lt;tran:encoding xmlns:tran="http://www.bea.com/wli/sb/transports"&gt;utf-8&lt;/tran:encoding&gt;
            &lt;http:client-host xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;1.2.3.4&lt;/http:client-host&gt;
            &lt;http:client-address xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;1.2.3.4&lt;/http:client-address&gt;
            &lt;http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http"&gt;POST&lt;/http:http-method&gt;
          &lt;/xml-fragment&gt;
       Payload =  
      &lt;s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"&gt;&lt;XMLHere/&gt;&lt;/s:Envelope&gt;
  </code></pre>
  
  <p>Here is my lexer:</p>
  
  <pre><code>lexer grammar LogLexer;
  
  options {filter=true;}
  
   /*------------------------------------------------------------------
   * LEXER RULES
   *------------------------------------------------------------------*/
  LOGDATE : DAY ' ' MONTH ' ' NUMDAY ' ' NUMTIME ' ' TIMEZONE ' ' NUMYEAR;
  
  METAINFO : '[' .* ']' ' [[' .* ']' .* ']' .* '-' .* '[OSB Tracing] Inbound request was received.';
  
  SERVICE_REF : 'Service Ref = ';
  
  URI : (SYMBOL | '/')+;
  
  ENVELOPE_TAG : '&lt;' ENVELOPE_TAGNAME .* '&gt;' .* '&lt;/' ENVELOPE_TAGNAME '&gt;';
  
  fragment
  ENVELOPE_TAGNAME : SYMBOL+ ':Envelope';
  
  fragment
  NUMTIME : NUM NUM ':' NUM NUM ':' NUM NUM;
  
  fragment
  TIMEZONE : SYMBOL SYMBOL SYMBOL;
  
  fragment
  DAY : 'Sun' | 'Mon' | 'Tue' | 'Wed' | 'Fri' | 'Sat';
  
  fragment
  MONTH :  'Sep' | 'Oct' | 'Nov' | 'Dec' | 'Feb' | 'Mar' | 'May' | 'Apr' | 'Jun' | 'Jul' | 'Aug';
  
  fragment
  NUMYEAR : NUM NUM NUM NUM;
  
  fragment
  NUMDAY : NUM NUM;
  
  fragment
  NUM : '0'..'9';
  
  fragment
  SYMBOL : ('a'..'z' | 'A'..'Z');
  </code></pre>
  
  <p>And here is the parser (not finished yet):</p>
  
  <pre><code>grammar LogParser;
  
  options {
  tokenVocab = OSBLogLexer;
  }
  
  @header {
      import java.util.List;
      import java.util.ArrayList;
  }
  
  parse 
      returns [List&lt;List&lt;String&gt;&gt; entries] 
      @init {
          $entries = new ArrayList&lt;List&lt;String&gt;&gt;();
      }
      : requestLogEntry+
      {
          $entries.add($requestLogEntry.logEntry);
      };
  
  requestLogEntry 
      returns [List&lt;String&gt; logEntry]
      @init {
          $logEntry = new ArrayList&lt;String&gt;();
      }
      : LOGDATE METAINFO .* serviceRef .* ENVELOPE_TAG
      {
          $logEntry.add($LOGDATE.getText());
          $logEntry.add($serviceRef.serviceURI);
          $logEntry.add($ENVELOPE_TAG.getText());
      };
  
  serviceRef 
      returns [String serviceURI] 
      : SERVICE_REF URI 
      {
          $serviceURI = $URI.getText();
      };
  </code></pre>
  
  <p>The problem is that it parses log incorrectly. My code does not ignore unwanted records, so I get invalid DATE value in resulting list: Tue Aug 28 09:39:09 MSD 2012 (the first one in the example) instead of Mon Sep 03 10:35:54 MSD 2012 (the correct one). Could anyone help me?</p>
  
  <p>Thanks in advance for your answers.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I have updated my code, but I get generation errors. Can't see what is wrong.</p>
  
  <p>Updated lexer:</p>
  
  <pre><code>lexer grammar LogLexer;
  
  options {
      filter=true;
  }
  
  TRASH : LOGDATE ' ' METAINFO (' ' | '\n')* { skip(); };
  
  LOGDATE : DAY ' ' MONTH ' ' NUMDAY ' ' NUMTIME ' ' TIMEZONE ' ' NUMYEAR;
  
  METAINFO : ('[' | ']' | SYMBOL | NUM | ' ' | SPECIAL)+;
  
  OSB_METAINFO : (' ' | '\n')* '[OSB Tracing] Inbound request was received.';
  
  SERVICE_REF : 'Service Ref = ';
  
  URI : (SYMBOL | '/')+;
  
  ENVELOPE_TAG : '&lt;' ENVELOPE_TAGNAME .* '&gt;' .* '&lt;/' ENVELOPE_TAGNAME '&gt;';
  
  fragment
  OSB_TRACING : '[OSB Tracing] Inbound request was received.';
  
  fragment
  ENVELOPE_TAGNAME : SYMBOL+ ':Envelope';
  
  fragment
  NUMTIME : NUM NUM ':' NUM NUM ':' NUM NUM;
  
  fragment
  TIMEZONE : SYMBOL SYMBOL SYMBOL;
  
  fragment
  DAY : 'Sun' | 'Mon' | 'Tue' | 'Wed' | 'Fri' | 'Sat';
  
  fragment
  MONTH :  'Sep' | 'Oct' | 'Nov' | 'Dec' | 'Feb' | 'Mar' | 'May' | 'Apr' | 'Jun' | 'Jul' | 'Aug';
  
  fragment
  NUMYEAR : NUM NUM NUM NUM;
  
  fragment
  NUMDAY : NUM NUM;
  
  fragment
  NUM : '0'..'9';
  
  fragment
  SYMBOL : ('a'..'z' | 'A'..'Z');
  
  fragment
  SPECIAL : ( ~'\n' | '\'' | '.' | '(' | ')' | '-');
  </code></pre>
  
  <p>Updated parser:</p>
  
  <pre><code>parser grammar LogParser;
  
  options {
      tokenVocab = LogLexer;
  }
  
  @header {
      import java.util.List;
      import java.util.ArrayList;
  }
  
  parse returns [List&lt;List&lt;String&gt;&gt; entries] 
      @init {
          $entries = new ArrayList&lt;List&lt;String&gt;&gt;();
      }
      : requestLogEntry+
      {
          $entries.add($requestLogEntry.logEntry);
      };
  
  requestLogEntry 
      returns [List&lt;String&gt; logEntry]
      @init {
          $logEntry = new ArrayList&lt;String&gt;();
      }
      :  LOGDATE ' ' METAINFO OSB_METAINFO .* serviceRef .* ENVELOPE_TAG
      {
          $logEntry.add($LOGDATE.getText());
          $logEntry.add($serviceRef.serviceURI);
          $logEntry.add($ENVELOPE_TAG.getText());
      };
  
  serviceRef 
      returns [String serviceURI] 
      : SERVICE_REF URI 
      {
          $serviceURI = $URI.getText();
      };
  </code></pre>
  
  <p>Lexer generation errors:</p>
  
  <pre><code>[14:18:12] error(204): LogLexer.g:56:21: duplicate token type '\'' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:28: duplicate token type '.' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:34: duplicate token type '(' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:40: duplicate token type ')' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:46: duplicate token type '-' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:21: duplicate token type '\'' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:28: duplicate token type '.' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:34: duplicate token type '(' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:40: duplicate token type ')' when collapsing subrule into set
  [14:18:12] error(204): LogLexer.g:56:46: duplicate token type '-' when collapsing subrule into set
  </code></pre>
  
  <p>Those errors seem to happen randomly and randomly dissappear (file rename). Also ANTLR generates another lexer from my parser file (this also happens randomly). I am using last avaliable ANTLR3 and ANTLRWorks on Windows 7 (x64).</p>