title: AWS Glue job throwing java.lang.OutOfMemoryError while writing to s3
body: <p>I have a glue job that reads from RDS and writes it to s3 in parquet format with partitions.</p>
  <p>Below is the script. While the data is loaded into the frame and the count, schema is printed in a log. It fails with an error while writing the parquet to s3.</p>
  <p>Also, I have mostly used spark's data frame since I was just testing glue out.</p>
  <p>The size of the data is around <code>150 GB</code></p>
  <pre><code>import json
  import sys
  
  from awsglue.context import GlueContext
  from awsglue.job import Job
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkContext
  from pyspark.sql.functions import col, udf, year, month
  from pyspark.sql.functions import to_json, struct
  
  args = getResolvedOptions(sys.argv, [&quot;JOB_NAME&quot;])
  
  sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args[&quot;JOB_NAME&quot;], args)
  
  POSTGRES_USER = &quot;postgres&quot;
  POSTGRES_PASSWORD = &quot;postgres&quot;
  POSTGRES_JDBC_URL = &quot;&lt;host&gt;&quot;
  
  df = spark.read.format(&quot;jdbc&quot;).option(&quot;url&quot;, POSTGRES_JDBC_URL) \
      .option(&quot;driver&quot;, &quot;org.postgresql.Driver&quot;).option(&quot;dbtable&quot;, &quot;log&quot;) \
      .option(&quot;user&quot;, POSTGRES_USER).option(&quot;password&quot;, POSTGRES_PASSWORD).load()
  
  
  print(&quot;Log Count : &quot;, df.count())
  print(&quot;Log Schema : &quot;, df.printSchema())
  
  df_with_year_and_month = df.withColumn(&quot;year&quot;, year(col(&quot;created_at&quot;))).withColumn(&quot;month&quot;, month(col(&quot;created_at&quot;)))
  
  print(&quot;Log count : &quot;, df_with_year_and_month.count())
  print(&quot;Log schema : &quot;, df_with_year_and_month.printSchema())
  
  df_with_year_and_month.write.partitionBy(&quot;year&quot;, &quot;month&quot;).parquet(&quot;s3a://datalake-bucket/slogs-parquet&quot;,mode=&quot;append&quot;)
  job.commit()
  </code></pre>