title: Pyspark reading csv array column in the middle
body: <p>I have a csv which has a column which is supposed to contain arrays. So, when I read this csv through pyspark.sql, it gives null values for all the columns after the array column. How can I fix this?</p>
  
  <p>CSV:
  <a href="https://i.stack.imgur.com/OR7tg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OR7tg.png" alt="enter image description here"></a></p>
  
  <p>So when I read it back, with this
  <code>test_file = ss.read.csv('s3://sk1911-mdf-mybucket/big_subset.csv', header=True)</code>
  and do a take, it gives me something like this. </p>
  
  <pre><code>|               22050|0.6479336219623286|0.48202282669567886|AR4PQ891187FB5CA9F|       40.76099|East Orange, NJ|       -74.20991|Dionne Warwick|0.7901756112133602|0.09891509433962263|109.34542924528301|0.555841121495327|110.11071338785048|         0.0|222.9024|         0.299|   0.0|   5|         0.762|  -15.28|   1|          0.613|Friends|             0.5946|     96.034035| 0.6345415430267063|   -18.31317210682493|                    (674,)|    -27.020842729970326|       (674, 12)|111.5988821364985|      (674, 12)|         (100,)|0.24906579485370342|SOWTLUT12A8C13BE83|          210.396|0.5210292056074767|110.2390154088785|116.866|             4|                    0.981|Remember Your Heart|1985|['brill building ...|
  | 'disco' 'soul ja...|              null|               null|              null|           null|           null|            null|          null|              null|               null|              null|             null|              null|        null|    null|          null|  null|null|          null|    null|null|           null|   null|               null|          null|               null|                 null|                      null|                   null|            null|             null|           null|           null|               null|              null|             null|              null|             null|   null|          null|                     null|               null|null|                null|
  | 'chill-out' 'ger...|              null|               null|              null|           null|           null|            null|          null|              null|               null|              null|             null|              null|        null|    null|          null|  null|null|  
  </code></pre>
  
  <p>The array is leaking into the next rows and pyspark isn't able to read the further rows. Please help.</p>
  
  <p>The records look something like this. </p>
  
  <pre><code>|analysis_sample_rate|artist_familiarity|  artist_hotttnesss|         artist_id|artist_latitude|artist_location|artist_longitude|   artist_name| artist_terms_freq|    bars_confidence|        bars_start| beats_confidence|       beats_start|danceability|duration|end_of_fade_in|energy|key|key_confidence|loudness|mode|mode_confidence|release|sections_confidence|sections_start|segments_confidence|segments_loudness_max|segments_loudness_max_time|segments_loudness_start|segments_pitches|   segments_start|segments_timbre|similar_artists|    song_hotttnesss|           song_id|start_of_fade_out| tatums_confidence|     tatums_start|  tempo|time_signature|time_signature_confidence|              title|year|        artist_terms|
  +--------------------+------------------+-------------------+------------------+---------------+---------------+----------------+--------------+------------------+-------------------+------------------+-----------------+------------------+------------+--------+--------------+------+---+--------------+--------+----+---------------+-------+-------------------+--------------+-------------------+---------------------+--------------------------+-----------------------+----------------+-----------------+---------------+---------------+-------------------+------------------+-----------------+------------------+-----------------+-------+--------------+-------------------------+-------------------+----+--------------------+
  |               22050|0.6479336219623286|0.48202282669567886|AR4PQ891187FB5CA9F|       40.76099|East Orange, NJ|       -74.20991|Dionne Warwick|0.7901756112133602|0.09891509433962263|109.34542924528301|0.555841121495327|110.11071338785048|         0.0|222.9024|         0.299|   0.0|  5|         0.762|  -15.28|   1|          0.613|Friends|             0.5946|     96.034035| 0.6345415430267063|   -18.31317210682493|                    (674,)|    -27.020842729970326|       (674, 12)|111.5988821364985|      (674, 12)|         (100,)|0.24906579485370342|SOWTLUT12A8C13BE83|          210.396|0.5210292056074767|110.2390154088785|116.866|             4|                    0.981|Remember Your Heart|1985|['brill building ...|
  </code></pre>