title: job aborted while writing parquet to s3 via glue jobs
body: <p>My code looks like below which consists of transformations:</p>
  <pre><code>dictionaryDf = spark.read.option(&quot;header&quot;, &quot;true&quot;).csv(
              &quot;s3://...../.csv&quot;)
  
          web_notif_data = fullLoad.cache()
          web_notif_data.persist(StorageLevel.MEMORY_AND_DISK)
          print(&quot;::::::data has been loaded::::::::::::&quot;)
          distinct_campaign_name = web_notif_data.select(
              trim(web_notif_data.campaign_name).alias(&quot;campaign_name&quot;)).distinct()
          web_notif_data.createOrReplaceTempView(&quot;temp&quot;)
          variablesList = Config.get('web', 'variablesListWeb')
          web_notif_data = spark.sql(variablesList)
          web_notif_data.persist(StorageLevel.MEMORY_AND_DISK)
          web_notif_data = web_notif_data.withColumn(&quot;camp&quot;, regexp_replace(&quot;campaign_name&quot;, &quot;_&quot;, &quot;&quot;))
          web_notif_data = web_notif_data.drop(&quot;campaign_name&quot;)
          web_notif_data = web_notif_data.withColumnRenamed(&quot;camp&quot;, &quot;campaign_name&quot;)
          web_notif_data = web_notif_data.withColumn(&quot;channel&quot;, lit(&quot;web_notification&quot;))
          web_notif_data.createOrReplaceTempView(&quot;data&quot;)
          campaignTeamWeb = Config.get('web', 'campaignTeamWeb')
          web_notif_data = spark.sql(campaignTeamWeb)
          web_notif_data.persist(StorageLevel.MEMORY_AND_DISK)
  
          distinct_campaign_name = distinct_campaign_name.withColumn(&quot;camp&quot;, F.regexp_replace(
              F.lower(F.trim(col(&quot;campaign_name&quot;))),
              &quot;[^a-zA-Z0-9]&quot;, &quot;&quot;))
          output_df3 = (
              distinct_campaign_name.withColumn(&quot;cname_split&quot;,
                                                F.explode(F.split(F.lower(F.trim(col(&quot;campaign_name&quot;))), &quot;_&quot;)))
                  .join(
                  dictionaryDf,
                  (
                          (
                                  (F.col(&quot;function&quot;) == &quot;contains&quot;) &amp;
                                  F.col(&quot;camp&quot;).contains(F.col(&quot;terms&quot;))
                          ) |
                          (
                                  (F.col(&quot;function&quot;) == &quot;match&quot;) &amp;
                                  F.col(&quot;campaign_name&quot;).contains(&quot;_&quot;) &amp;
                                  (F.col(&quot;cname_split&quot;) == F.col(&quot;terms&quot;))
                          )
                  ),
                  &quot;left&quot;
              )
                  .withColumn(
                  &quot;empty_is_other&quot;,
                  F.when(
                      (
                              F.col(&quot;product&quot;).isNull() &amp;
                              F.col(&quot;product_category&quot;).isNull()
                      ),
                      &quot;other&quot;
                  )
              )
                  .withColumn(
                  &quot;rn&quot;,
                  F.row_number().over(
                      Window.partitionBy(&quot;campaign_name&quot;)
                          .orderBy(
                          F.when(
                              F.col(&quot;function&quot;).isNull(), 3
                          ).when(
                              F.col(&quot;function&quot;) == &quot;match&quot;, 2
                          ).otherwise(1),
                          F.length(F.col(&quot;terms&quot;)).desc(),
                          F.col(&quot;product&quot;).isNull()
                      )
                  )
              )
                  .filter(&quot;rn=1&quot;)
                  .select(
                  &quot;campaign_name&quot;,
                  F.coalesce(&quot;product&quot;, &quot;empty_is_other&quot;).alias(&quot;prod&quot;),
                  F.coalesce(&quot;product_category&quot;, &quot;empty_is_other&quot;).alias(&quot;prod_cat&quot;),
              )
                  .na.fill(&quot;&quot;)
          )
          print(&quot;:::::::::::transformations have been done finally::::::::::::&quot;)
          web_notif_data1 = web_notif_data  # Just taking the backup of DF in case something goes wrong
          web_notif_data = web_notif_data.drop(&quot;campaign_name&quot;)
          web_notif_data = web_notif_data.withColumnRenamed(&quot;temp_campaign_name&quot;, &quot;campaign_name&quot;)
          veryFinalDF = web_notif_data.join(output_df3, &quot;campaign_name&quot;, &quot;left_outer&quot;)
          # veryFinalDF.show(truncate=False)
          veryFinalDF.write.mode(&quot;overwrite&quot;).parquet(aggregatedPath)
          print(&quot;::::final data have been written successfully::::::&quot;)
  </code></pre>
  <p>where <code>fullLoad</code> is the data frame that reads from the Redshift table.This code works fine on <code>0.2 Million</code> records. However, in production for 15 days, data could be around a <code>minimum</code> of <code>25 Million</code> records. I don't know the size since the data is stored in the redshift table and we are reading from it and then processing the data. I am running this code via <code>Glue</code> jobs and it gets stuck on the last line i.e while writing the data as parquet. It gives me the below error:</p>
  <p><a href="https://i.stack.imgur.com/p6mJd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/p6mJd.png" alt="enter image description here" /></a></p>
  <p>I tried running it with <code>30</code> executors. It takes around <code>20 mins</code> to load the data from <code>Redshift</code> into the <code>fullLoad</code> dataframe. What else can be done to avoid this error? I am new to AWS and Glue jobs.</p>