Data, Data Engineer, Open Source, Spark, Kubernetes.
many small files — merge them our job may be launching too many API calls/requests — implement backoff we don’t have an optimal partition strategy — Fix partitions. Error 6 The executor with id 98 was deleted by a user or the framework. This issue appears intermittently while using dynamic