title: Storing pandas DataFrames in SQLAlchemy models
body: <p>I'm building a flask application that allows users to upload CSV files (with varying columns), preview uploaded files, generate summary statistics, perform complex transformations/aggregations (sometimes via Celery jobs), and then export the modified data. The uploaded file is being read into a pandas DataFrame, which allows me to elegantly handle most of the complicated data work.</p>
  
  <p>I'd like these DataFrames along with associated metadata (time uploaded, ID of user uploading the file, etc.) to persist and be available for multiple users to pass around to various views. However, I'm not sure how best to incorporate the data into my SQLAlchemy models (I'm using PostgreSQL on the backend).</p>
  
  <p>Three approaches I've considered:</p>
  
  <ul>
  <li>Cramming the DataFrame into a <code>PickleType</code> and storing it directly in the DB. This seems to be the most straightforward solution, but means I'll be sticking large binary objects into the database.</li>
  <li>Pickling the DataFrame, writing it to the filesystem, and storing the path as a string in the model. This keeps the database small but adds some complexity when backing up the database and allowing users to do things like delete previously uploaded files.</li>
  <li>Converting the DataFrame to JSON (<code>DataFrame.to_json()</code>) and storing it as a <code>json</code> type (maps to PostgreSQL's <code>json</code> type). This adds the overhead of parsing JSON each time the DataFrame is accessed, but it also allows the data to be manipulated directly via <a href="http://www.postgresql.org/docs/9.3/static/functions-json.html">PostgreSQL JSON operators</a>.</li>
  </ul>
  
  <p>Given the advantages and drawbacks of each (including those I'm unaware of), is there a preferred way to incorporate pandas DataFrames into SQLAlchemy models?</p>