title: pyLDAvis visualization of pyspark generated LDA model
body: <p>Does anyone have an example of data visualization of an LDA model trained using the PySpark library (specifically using <a href="https://github.com/bmabey/pyLDAvis" rel="noreferrer">pyLDAvis</a>)? I've seen a lot of examples for GenSim and other libraries but not PySpark. Specifically I'm wondering what to pass into the <code>pyLDAvis.prepare()</code> function and how to get it from my lda model.
  Here is my code:</p>
  
  <pre><code> from pyspark.mllib.clustering import LDA, LDAModel
   from pyspark.mllib.feature import IDF
   from pyspark.ml.feature import CountVectorizer
   from pyspark.mllib.linalg import Vectors
  
   vectorizer = CountVectorizer(inputCol="filtered1", outputCol="features").fit(filtered_final)
   countVectors = vectorizer.transform(filtered_final).select("status_id", "features")
   countVectors.show()
   frequencyVectors = countVectors.rdd.map(lambda vector: vector[1])
   frequencyDenseVectors = frequencyVectors.map(lambda vector: Vectors.dense(vector))
   idf = IDF().fit(frequencyDenseVectors)
   print('fitting complete')
   tfidf = idf.transform(frequencyDenseVectors)
   print("tf idf complete")
   #prepare corpus for LDA
   corpus = tfidf.map(lambda x: [1, x]).cache()
   #train LDA
   ldaModel = LDA.train(corpus, k = 15, maxIterations=100, optimizer="online", docConcentration=2.0, topicConcentration=3.0)
   print("lda model complete")
  </code></pre>