title: Convert data types in JSON from Kafka Sparkstreaming
body: <p>I have a JSON which I am reading from a kafka topic using spark streaming</p>
  <pre><code>{&quot;COUNTRY_REGION&quot;: &quot;United States&quot;,  &quot;GROCERY_AND_PHARMACY_CHANGE_PERC&quot;: &quot;-7&quot;, &quot;PARKS_CHANGE_PERC&quot;: &quot;\\\\N&quot;,  &quot;LAST_UPDATE_DATE&quot;: &quot;05:31.7&quot;}
  </code></pre>
  <p>I understand that we would first need to create a schema which I have done here and parse the input json which we got from Kafka i.e. value field via the from_json function.</p>
  <pre><code>schema = StructType([ 
      StructField(&quot;COUNTRY_REGION&quot;,StringType(),True), 
      StructField(&quot;PROVINCE_STATE&quot;,StringType(),True),
      StructField(&quot;ISO_3166_1&quot;,StringType(),True), 
      StructField(&quot;ISO_3166_2&quot;, StringType(), True), 
      StructField(&quot;DATE&quot;, DateType(), True), 
      StructField(&quot;GROCERY_AND_PHARMACY_CHANGE_PERC&quot;, IntegerType(), True),
      StructField(&quot;PARKS_CHANGE_PERC&quot;, IntegerType(), True), 
      StructField(&quot;RESIDENTIAL_CHANGE_PERC&quot;, IntegerType(), True), 
      StructField(&quot;RETAIL_AND_RECREATION_CHANGE_PERC&quot;, IntegerType(), True),                    
      StructField(&quot;TRANSIT_STATIONS_CHANGE_PERC&quot;, IntegerType(), True),  
      StructField(&quot;WORKPLACES_CHANGE_PERC&quot;, IntegerType(), True), 
      StructField(&quot;LAST_UPDATE_DATE&quot;, DateType(), True),
      StructField(&quot;LAST_REPORTED_FLAG&quot;, BooleanType(), True),
      StructField(&quot;SUB_REGION_2&quot;, StringType(), True),
    ])
  
  json_edit = df.select (from_json(&quot;value&quot;,schema).alias(&quot;json&quot;))
  </code></pre>
  <p>However, I realise the <code>GROCERY_AND_PHARMACY_CHANGE_PERC</code>, <code>PARKS_CHANGE_PERC</code> and <code>LAST_UPDATE_DATE</code> become null.</p>
  <pre><code>display(json_edit)
  
  {&quot;COUNTRY_REGION&quot;: &quot;United States&quot;,  &quot;GROCERY_AND_PHARMACY_CHANGE_PERC&quot;: null, &quot;PARKS_CHANGE_PERC&quot;: null, &quot;LAST_UPDATE_DATE&quot;: null}
  </code></pre>
  <p>I realized that it's because of the original JSON, for example <code>&quot;GROCERY_AND_PHARMACY_CHANGE_PERC&quot;: &quot;-7&quot;</code> when it should be <code>&quot;GROCERY_AND_PHARMACY_CHANGE_PERC&quot;: -7</code>.</p>
  <p>Is there any way to convert the string to a double/int before I parse it into the schema?</p>
  <p>Thank you!</p>