title: how to parse html files in pyspark and use Beautifulsoup?
body: <p>I have a very difficult situation:
  I need to parse a bunch of html files in <strong>pyspark</strong>, but I still want to use <strong>BeautifulSoup</strong> to parse html files. The dilemma is:</p>
  
  <ol>
  <li>If I saved these html files in <strong>HDFS</strong>, and use pyspark to read html files in, I can only read them as <strong>RDD</strong>, but I cannot pass RDD as the input parameter in BeautifulSoup;</li>
  <li>If I saved these html files in local, and use BeautifulSoup to parse html files, the power of pyspark is not been used. </li>
  </ol>
  
  <p>How can I do it? </p>