title: Parsing HTML issues with Apache Tika
body: <p>I am crawling a webpage and after crawling it extract all the links from that webpage and then I am trying to parse all the url using Apache Tika and BoilerPipe by using below code so for some url it is parsing very well but for some I get error like this. And it shows some error on HTMLParser.java: line number 102. This is line number 102 in HTMLParser.java</p>
  
  <pre><code>String parsedText = tika.parseToString(htmlStream, md);
  </code></pre>
  
  <p>I have provided the HTMLParse code also.</p>
  
  <pre><code>org.apache.tika.exception.TikaException: TIKA-198: Illegal IOException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@67c28a6a
          at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:203)
          at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:197)
          at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:135)
          at org.apache.tika.Tika.parseToString(Tika.java:357)
          at edu.uci.ics.crawler4j.crawler.HTMLParser.parse(HTMLParser.java:102)
          at edu.uci.ics.crawler4j.crawler.WebCrawler.handleHtml(WebCrawler.java:227)
          at edu.uci.ics.crawler4j.crawler.WebCrawler.processPage(WebCrawler.java:299)
          at edu.uci.ics.crawler4j.crawler.WebCrawler.run(WebCrawler.java:118)
          at java.lang.Thread.run(Unknown Source)
  Caused by: java.util.zip.ZipException: invalid block type
          at java.util.zip.InflaterInputStream.read(Unknown Source)
          at java.util.zip.ZipInputStream.read(Unknown Source)
          at java.io.FilterInputStream.read(Unknown Source)
          at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource$FakeZipEntry.&lt;init&gt;(ZipInputStreamZipEntrySource.java:114)
          at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.&lt;init&gt;(ZipInputStreamZipEntrySource.java:55)
          at org.apache.poi.openxml4j.opc.ZipPackage.&lt;init&gt;(ZipPackage.java:82)
          at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:220)
          at org.apache.poi.extractor.ExtractorFactory.createExtractor(ExtractorFactory.java:152)
          at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:65)
          at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:67)
          at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:197)
          ... 8 more
  </code></pre>
  
  <p>This is my HTMLParser.java file-</p>
  
  <pre><code>    public void parse(String htmlContent, String contextURL) {
  
          InputStream htmlStream = null;
          text = null;
          title = null;
          metaData = new HashMap&lt;String, String&gt;();
  
          urls = new HashSet&lt;String&gt;();
          char[] chars = htmlContent.toCharArray();
  
          bulletParser.setCallback(textExtractor);
          bulletParser.parse(chars);
  
          try {
              text = articleExtractor.getText(htmlContent);
          } catch (BoilerpipeProcessingException e) {
              e.printStackTrace();
          }
  
          if (text == null){
              text = textExtractor.text.toString().trim(); 
          }
  
          title = textExtractor.title.toString().trim();
          try {
              Metadata md = new Metadata();
              String utfHtmlContent = new String(htmlContent.getBytes(),"UTF-8");
              htmlStream = new ByteArrayInputStream(utfHtmlContent.getBytes());
      //The below line is at the line number 102 according to error above
                  String parsedText = tika.parseToString(htmlStream, md);
                  //very unlikely to happen
                  if (text == null){
                      text = parsedText.trim();
                  }
                  processMetaData(md);
              } catch (Exception e) {
                  e.printStackTrace();
              } finally {
                  IOUtils.closeQuietly(htmlStream);
              }
              bulletParser.setCallback(linkExtractor);
              bulletParser.parse(chars);
              Iterator&lt;String&gt; it = linkExtractor.urls.iterator();
  
              String baseURL = linkExtractor.base();
              if (baseURL != null) {
                  contextURL = baseURL;
              }
  
              int urlCount = 0;
              while
  
   (it.hasNext()) {
              String href = it.next();
              href = href.trim();
              if (href.length() == 0) {
                  continue;
              }
              String hrefWithoutProtocol = href.toLowerCase();
              if (href.startsWith("http://")) {
                  hrefWithoutProtocol = href.substring(7);
              }
              if (hrefWithoutProtocol.indexOf("javascript:") &lt; 0
                      &amp;&amp; hrefWithoutProtocol.indexOf("@") &lt; 0) {
                  URL url = URLCanonicalizer.getCanonicalURL(href, contextURL);
                  if (url != null) {
                      urls.add(url.toExternalForm());
                      urlCount++;
                      if (urlCount &gt; MAX_OUT_LINKS) {
                          break;
                      }   
                  }               
              }
          }
      }
  </code></pre>
  
  <p>Any suggestions will be appreciated.</p>