Answers.Body: <p>There are different possible ways to handle huge datasets:</p>
  <ol>
  <li><p>If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in <a href="https://towardsdatascience.com/why-and-how-to-use-pandas-with-large-data-9594dda2ea4c" rel="nofollow noreferrer">the article <code>Why and How to Use Pandas with Large Data</code></a>, section <code>1. Read CSV file data in chunk size</code>. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.</p>
  </li>
  <li><p>If the data is really big, probably it's better to store and process it by multiple computers using special software, A specific tool depends on what you want to do with the data:</p>
  </li>
  </ol>
  <ul>
  <li>2.1. You can even keep using Pandas: there is <a href="https://stackoverflow.com/a/47426521/5308802">an extension named Dask</a> which wraps interfaces of Python iterables, NumPy, and Pandas to run on multiple machines. Also, there are other similar tools.</li>
  <li>2.2. A more independent yes simple approach is to use some analytical, distributed DBMS like Google BigQuery or Cassandra.</li>
  <li>2.3. And if you need something more powerful and complex, MapReduce systems like Apache Hadoop, Spark can be utilised. Moreover, there are many <a href="https://livebook.manning.com/book/machine-learning-in-action/chapter-15/135" rel="nofollow noreferrer">articles</a> and <a href="https://papers.nips.cc/paper/2006/file/77ee3bc58ce560b86c2b59363281e914-Paper.pdf" rel="nofollow noreferrer">scientific papers</a> that describes usage of MapReduce for the machine learning.</li>
  </ul>
  <p>So, as always, it is a matter of your opportunities and your aims of dealing with the data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-03T06:44:32.197Z
Answers.LastActivityDate: 2020-11-18T06:30:59.810Z
Answers.LastEditDate: 2020-11-18T06:30:59.810Z
Answers.UserDisplayName: AivanF.
Body: <p>Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem to be meant for either pure text or pure numerical data.</p>
  
  <p>There are those that can handle data with the dual data types, but those are generally not flexible for wrangling. For example, for pickle you have to load the entire thing into memory if you want to wrangle anything. You can just append directly to the disk like you can with hdf5, which can be very helpful for huge datasets which can not be all loaded into memory?</p>
  
  <p>Also, any alternatives to Pandas for wrangling Huge datasets? Sometimes you can't load all the data into Pandas without causing a memory crash.</p>
  
Tags: <natural-language-processing><datasets><data-science><structured-data><data-mining>
Title: What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?
UserDisplayName: SantoshGupta7
ThreadText: # What data formats/pipelining are best to store and wrangle data which contains both text and float vectors?
  
  (Asked by: username_0 on 2019-05-02)
  
  Often in NLP project the data points contain both text and float embeddings, and it's very tricky to deal with. CSVs take up a ton of memory and are slow to load. But most the other data formats seem to be meant for either pure text or pure numerical data.
  
  There are those that can handle data with the dual data types, but those are generally not flexible for wrangling. For example, for pickle you have to load the entire thing into memory if you want to wrangle anything. You can just append directly to the disk like you can with hdf5, which can be very helpful for huge datasets which can not be all loaded into memory?
  
  Also, any alternatives to Pandas for wrangling Huge datasets? Sometimes you can't load all the data into Pandas without causing a memory crash.
  
  **Tags:** natural-language-processing, datasets, data-science, structured-data, data-mining
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-03. Score: 2 | Accepted answer
  
  There are different possible ways to handle huge datasets:
  
  1. If the data is too big to be fully uploaded to RAM, you can iterate over it in Pandas. You can find a brief explanation in [the article `Why and How to Use Pandas with Large Data`](https://towardsdatascience.com/why-and-how-to-use-pandas-with-large-data-9594dda2ea4c), section `1. Read CSV file data in chunk size`. Or add more RAM (or use powerful server hardware), if you want to continue using single machine.
  2. If the data is really big, probably it's better to store and process it by multiple computers using special software, A specific tool depends on what you want to do with the data:
  
  * 2.1. You can even keep using Pandas: there is [an extension named Dask](https://stackoverflow.com/a/47426521/5308802) which wraps interfaces of Python iterables, NumPy, and Pandas to run on multiple machines. Also, there are other similar tools.
  * 2.2. A more independent yes simple approach is to use some analytical, distributed DBMS like Google BigQuery or Cassandra.
  * 2.3. And if you need something more powerful and complex, MapReduce systems like Apache Hadoop, Spark can be utilised. Moreover, there are many [articles](https://livebook.manning.com/book/machine-learning-in-action/chapter-15/135) and [scientific papers](https://papers.nips.cc/paper/2006/file/77ee3bc58ce560b86c2b59363281e914-Paper.pdf) that describes usage of MapReduce for the machine learning.
  
  So, as always, it is a matter of your opportunities and your aims of dealing with the data.
  
  
  ---
  
  