title: How to remove a comma from a token using a Custom TokenFilter in Lucene.net
body: <p>I have a custom tokenfilter setup to parse keywords such as</p>
  
  <pre><code>oracle,java,sybase,vb.net etc. 
  </code></pre>
  
  <p>into</p>
  
  <pre><code>oracle java sybase vb.net
  </code></pre>
  
  <p>and it's working fine, however one of the test docs has the following text</p>
  
  <pre><code>,oracle java,sybase,unix
  </code></pre>
  
  <p>and i'm trying to remove the leading COMMA from </p>
  
  <pre><code>,oracle
  </code></pre>
  
  <p>using the code below</p>
  
  <pre><code>    public override bool IncrementToken()
      {
          if (!input.IncrementToken())
              return false;
  
  
          char[] buffer = termAtt.TermBuffer();
          int bufferLength = termAtt.TermLength();
  
  ...
          else if (bufferLength &gt; 1 &amp;&amp; buffer[0] == ',')
          {
              // strip the starting , off !
              offsetAtt.SetOffset(offsetAtt.StartOffset + 1, offsetAtt.EndOffset);
          // where offsetAtt = AddAttribute&lt;IOffsetAttribute&gt;();
          }
          ...
  
          return true;
  
      }
  </code></pre>
  
  <p>however that does not remove the comma</p>
  
  <p>any help on how to make this work ?</p>
  
  <p>thanks</p>