title: How to invalidate metadata for Impala using spark?
body: <p>I'm using PySpark to inset data into an empty table at first, but then I will have to automate the process. Using PySpark, how do I invalidate the metadata or refresh the data so that it can be read properly in Impala?</p>
  <p>Here is a sample of my code:</p>
  <pre><code>spark.sql(&quot;&quot;&quot;
  select
   gps_data_adj.trip_duration
   , gps_data_adj.geometry
   , trip_summary.TRIP_HAVERSINE_DISTANCE
   , trip_summary.TRIP_GPS_DURATION
   , gps_data_adj.HAVERSINE_DISTANCE
   , gps_data_adj.GPS_INTERVAL
   , gps_data_adj.HAVERSINE_DISTANCE/trip_summary.TRIP_HAVERSINE_DISTANCE AS HAVERSINE_DISTANCE_FRACTION
   , gps_data_adj.GPS_INTERVAL/trip_summary.TRIP_GPS_DURATION AS GPS_INTERVAL_FRACTION
   , (gps_data_adj.HAVERSINE_DISTANCE/trip_summary.TRIP_HAVERSINE_DISTANCE)*gps_data_adj.trip_distance_travelled AS HAVERSINE_DISTANCE_ADJ
   , (gps_data_adj.GPS_INTERVAL/trip_summary.TRIP_GPS_DURATION)*gps_data_adj.trip_duration AS GPS_INTERVAL_ADJ
      FROM
          gps_data_adj
      INNER JOIN
          (
              SELECT
                  trip_id 
                  , sum(COSINES_DISTANCE) as TRIP_COSINES_DISTANCE
                  , sum(HAVERSINE_DISTANCE) as TRIP_HAVERSINE_DISTANCE
                  , sum(GPS_INTERVAL) AS TRIP_GPS_DURATION
              FROM
                  gps_data_adj
              GROUP BY
                  trip_id
          ) trip_summary
  on gps_data_adj.trip_id = trip_summary.trip_id
  &quot;&quot;&quot;).write.format('parquet').mode('append').insertInto('driving_data_TEST')
  </code></pre>