Memory Allocation of Python Sub-Processes Within Pyspark Driver
Python driver memory is not accounted in spark.driver.memory (sames as spark.executor.memory in case of workers). spark.*.memory settings are used only for on-heap Java memory allocation.
All other forms of memory allocation (off-heap, native memory like in case of Python and such) are covered by spark.driver.memoryOverhead and equivalent settings.
When creating new Python processes within PySpark driver (e.g. with JobLib or other multi-processing library), do those processes share the Spark driver memory, or are they allocated memory outside of that PySpark driver JVM?