title: What analyzer should I use for a URL in lucene.net?
body: <p>I'm having problems getting a simple URL to tokenize properly so that you can search it as expected.</p>
  
  <p>I'm indexing "<a href="http://news.bbc.co.uk/sport1/hi/football/internationals/8196322.stm" rel="noreferrer">http://news.bbc.co.uk/sport1/hi/football/internationals/8196322.stm</a>" with the StandardAnalyzer and it is tokenizing the string as the following (debug output):</p>
  
  <pre><code>(http,0,4,type=&lt;ALPHANUM&gt;)
  (news.bbc.co.uk,7,21,type=&lt;HOST&gt;)
  (sport1/hi,22,31,type=&lt;NUM&gt;)
  (football,32,40,type=&lt;ALPHANUM&gt;)
  (internationals/8196322.stm,41,67,type=&lt;NUM&gt;)
  </code></pre>
  
  <p>In general it looks good, http itself, then the hostname but the issue seems to come with the forward slashes. Surely it should consider them as seperate words?</p>
  
  <p>What do I need to do to correct this?</p>
  
  <p>Thanks</p>
  
  <p>P.S. I'm using Lucene.NET but I really don't think it makes much of a difference with regards to the answers.</p>