title: AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks'>
body: <p>I was using pyspark on AWS EMR (4 r5.xlarge as 4 workers, each has one executor and 4 cores), and I got <code>AttributeError: Can't get attribute 'new_block' on &lt;module 'pandas.core.internals.blocks'</code>. Below is a snippet of the code that threw this error:</p>
  <pre><code>search =  SearchEngine(db_file_dir = &quot;/tmp/db&quot;)
  conn = sqlite3.connect(&quot;/tmp/db/simple_db.sqlite&quot;)
  pdf_ = pd.read_sql_query('''select  zipcode, lat, lng, 
                          bounds_west, bounds_east, bounds_north, bounds_south from 
                          simple_zipcode''',conn)
  brd_pdf = spark.sparkContext.broadcast(pdf_) 
  conn.close()
  
  
  @udf('string')
  def get_zip_b(lat, lng):
      pdf = brd_pdf.value 
      out = pdf[(np.array(pdf[&quot;bounds_north&quot;]) &gt;= lat) &amp; 
                (np.array(pdf[&quot;bounds_south&quot;]) &lt;= lat) &amp; 
                (np.array(pdf['bounds_west']) &lt;= lng) &amp; 
                (np.array(pdf['bounds_east']) &gt;= lng) ]
      if len(out):
          min_index = np.argmin( (np.array(out[&quot;lat&quot;]) - lat)**2 + (np.array(out[&quot;lng&quot;]) - lng)**2)
          zip_ = str(out[&quot;zipcode&quot;].iloc[min_index])
      else:
          zip_ = 'bad'
      return zip_
  
  df = df.withColumn('zipcode', get_zip_b(col(&quot;latitude&quot;),col(&quot;longitude&quot;)))
  </code></pre>
  <p>Below is the traceback, where line 102, in get_zip_b refers to <code>pdf = brd_pdf.value</code>:</p>
  <pre><code>21/08/02 06:18:19 WARN TaskSetManager: Lost task 12.0 in stage 7.0 (TID 1814, ip-10-22-17-94.pclc0.merkle.local, executor 6): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py&quot;, line 605, in main
      process()
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py&quot;, line 597, in process
      serializer.dump_stream(out_iter, outfile)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py&quot;, line 223, in dump_stream
      self.serializer.dump_stream(self._batched(iterator), stream)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py&quot;, line 141, in dump_stream
      for obj in iterator:
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/serializers.py&quot;, line 212, in _batched
      for item in iterator:
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py&quot;, line 450, in mapper
      result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py&quot;, line 450, in &lt;genexpr&gt;
      result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/worker.py&quot;, line 90, in &lt;lambda&gt;
      return lambda *a: f(*a)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/util.py&quot;, line 121, in wrapper
      return f(*args, **kwargs)
    File &quot;/mnt/var/lib/hadoop/steps/s-1IBFS0SYWA19Z/Mobile_ID_process_center.py&quot;, line 102, in get_zip_b
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py&quot;, line 146, in value
      self._value = self.load_from_path(self._path)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py&quot;, line 123, in load_from_path
      return self.load(f)
    File &quot;/mnt/yarn/usercache/hadoop/appcache/application_1627867699893_0001/container_1627867699893_0001_01_000009/pyspark.zip/pyspark/broadcast.py&quot;, line 129, in load
      return pickle.load(file)
  AttributeError: Can't get attribute 'new_block' on &lt;module 'pandas.core.internals.blocks' from '/mnt/miniconda/lib/python3.9/site-packages/pandas/core/internals/blocks.py'&gt;
  </code></pre>
  <p>Some observations and thought process:</p>
  <p>1, After doing some search online, the AttributeError in pyspark seems to be caused by mismatched pandas versions between driver and workers?</p>
  <p>2, But I ran the same code on two different datasets, one worked without any errors but the other didn't, which seems very strange and undeterministic, and it seems like the errors may not be caused by mismatched pandas versions. Otherwise, neither two datasets would succeed.</p>
  <p>3, I then ran the same code on the successful dataset again, but this time with different spark configurations: setting spark.driver.memory from 2048M to 4192m, and it threw AttributeError.</p>
  <p>4, In conclusion, I think the AttributeError has something to do with driver. But I can't tell how they are related from the error message, and how to fix it: AttributeError: Can't get attribute 'new_block' on &lt;module 'pandas.core.internals.blocks'.</p>