Data, Data Engineer, Open Source, Spark, Kubernetes.
the problem we can add a new path for the temp directory by setting java.io.tmpdir. PATH_TO_TEMP=hdfs://path/on/hdfs/for/tmp # and in spark submit add the following --conf spark.driver.extraJavaOptions=-Djava.io.tmpdir=${PATH_TO_TEMP} Error 4 CFUNCTYPE(c_int)(lambda: None) MemoryError PYTHONPATH