title: Fetch all values irrespective of keys from a column of JSON type in a Spark dataframe using Spark with scala
body: <p>I was trying to load a TSV file containing some metadata on movies using Spark, this TSV file contains genre info on movies in JSON format [ The last column in every row ]</p>
  
  <p><strong>Sample File</strong></p>
  
  <pre><code>975900  /m/03vyhn   Ghosts of Mars  2001-08-24  14010832    98.0    {"/m/02h40lc": "English Language"}  {"/m/09c7w0": "United States of America"}   {"/m/01jfsb": "Thriller", "/m/06n90": "Science Fiction", "/m/03npn": "Horror", "/m/03k9fj": "Adventure", "/m/0fdjb": "Supernatural", "/m/02kdv5l": "Action", "/m/09zvmj": "Space western"}
  3196793 /m/08yl5d   Getting Away with Murder: The JonBenét Ramsey Mystery   2000-02-16      95.0    {"/m/02h40lc": "English Language"}  {"/m/09c7w0": "United States of America"}   {"/m/02n4kr": "Mystery", "/m/03bxz7": "Biographical film", "/m/07s9rl0": "Drama", "/m/0hj3n01": "Crime Drama"}
  </code></pre>
  
  <p><strong>I had tried the below code which enables me to access a particular value from the genre JSON</strong></p>
  
  <pre><code>val ss = SessionCreator.createSession("DataCleaning", "local[*]")//helper function creates a spark session and returns it
  val headerInfoRb = ResourceBundle.getBundle("conf.headerInfo")
  val movieDF = DataReader.readFromTsv(ss, "D:/Utility/Datasets/MovieSummaries/movie.metadata.tsv")
                  .toDF(headerInfoRb.getString("metadataReader").split(',').toSeq:_*)//Datareader.readFromTsv is a helper function to read TSV file ,takes sparkSession and file path as input to resurn a dataframe, which uses sparkSession's read function 
  
  movieDF.select("wiki_mv_id","mv_nm","mv_genre")
                  .withColumn("genre_frmttd", get_json_object(col("mv_genre"), "$./m/02kdv5l"))
                  .show(1,false)
  </code></pre>
  
  <p><strong>Output</strong></p>
  
  <pre><code>+----------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
  |wiki_mv_id|mv_nm         |mv_genre                                                                                                                                                                                  |genre_frmttd|
  +----------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
  |975900    |Ghosts of Mars|{"/m/01jfsb": "Thriller", "/m/06n90": "Science Fiction", "/m/03npn": "Horror", "/m/03k9fj": "Adventure", "/m/0fdjb": "Supernatural", "/m/02kdv5l": "Action", "/m/09zvmj": "Space western"}|Action      |
  +----------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+
  only showing top 1 row
  </code></pre>
  
  <p><strong>I want genre_frmttd column in the way shown below for every row in the data Frame [ below snippet is for the first sample row ]</strong></p>
  
  <pre><code>[Thriller,Fiction,Horror,Adventure,Supernatural,Action,Space Western]
  </code></pre>
  
  <p>I am bit of a rookie in scala and spark, do suggest some way to list out the values</p>