title: Write Pandas DataFrame To S3 as Pickle
body: <p>Here are my requirements.</p>
  <ul>
  <li>Upload a pandas dataframe to AWS S3 as a pickle file</li>
  <li>Due to environment reasons, boto3 must be used and alternatives such as s3fs are not an option</li>
  <li>Data must exist in memory, and writing to temporary files is not possible</li>
  </ul>
  <p>I created the following simple function that uploads a Pandas dataframe to s3 as a csv:</p>
  <pre><code>def df_to_s3_csv(df, filename, sep=','):
      s3 = boto3.resource('s3')
      buffer = io.StringIO()
      df.to_csv(buffer, sep=sep, index=False)
      s3.Object(s3bucket, f'{s3_upload_path}/{filename}').put(Body=buffer.getvalue())
  </code></pre>
  <p>This function works fine and does what it is supposed to. For the pickle file, I created the following function in a similar manner:</p>
  <pre><code>def df_to_s3_pckl(df, filename):
      s3 = boto3.resource('s3')
      buffer = io.BytesIO()
      df.to_pickle(buffer)
      buffer.seek(0)
      obj = s3.Object(s3bucket, f'{s3_upload_path}/{filename}')
      obj.put(Body=buffer.getvalue())
  </code></pre>
  <p>I tried this function with and without the <code>seek</code> portion and either way it throws the following error: <code>ValueError: I/O operation on closed file.</code></p>
  <p>Looking further into the issue, I found that <code>buffer</code> is considered <code>closed</code> as soon as <code>df.to_pickle</code> is called. This is reproducible by issuing these commands:</p>
  <pre><code>buffer = io.BytesIO()
  df.to_pickle(buffer)
  print(buffer.closed)
  </code></pre>
  <p>The above prints <code>True</code>. It appears that the <code>BytesIO</code> buffer is closed by <code>to_pickle</code> and therefore its data cannot be referenced. How can this issue be resolved, or is there an alternative that meets my requirements? I've found several questions on SO about how to upload to S3 using boto3, but nothing regarding how to upload pickle files created by Pandas using BytesIO buffers.</p>
  <p>Here is a minimal reproducible example of the underlying issue:</p>
  <pre><code>import pandas as pd
  import numpy as np
  import io
  df = pd.DataFrame(np.random.randint(0,100,size=(4,4)))                                                                                   
  buffer = io.BytesIO()
  df.to_pickle(buffer)
  print(buffer.closed)
  </code></pre>