Answers.Body: <p>is your data stored in raw ASCII text, like a CSV file?</p>
  
  <p>Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.</p>
  
  <p>There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.</p>
  
  <p>Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.</p>
  
  <p>In some cases, you may need to resort to a big data platform.
  That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T10:56:03.517Z
Answers.Text: My data is in CSV having two columns, `img_path` and `label`. Image_path points to an actual image on which I have to train my CNN Model. So loading millions of images in memory is the main issue.
Answers.UserDisplayName: Gaurav Gandhi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T14:18:45.267Z
Answers.Text: how did u label your images ? did u use any tool or software ?.
Answers.UserDisplayName: Varshitha N S
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T17:50:01.357Z
Answers.LastActivityDate: 2019-08-24T17:50:01.357Z
Answers.UserDisplayName: Varshitha N S
Body: <p>I am in the process of collecting a huge dataset of Human poses captured images to create a model to classify poses.</p>
  
  <p>My question is how will I be able to train on this massive dataset? I have multiple GPUs and Multiple machines access (Also have GCP).</p>
  
  <p>What would be the best way to train on such huge dataset?</p>
  
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-22T02:52:13.460Z
Comments.Text: Welcome to Ai Community. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. Minimal, complete, verifiable example applies here. We cannot effectively help you until you post a sneak peak of  what you have tried out and accurately specify the problem.Sometimes,we leave out questions unanswered due to lack of a brief but descriptive summary of the problem as the title of the question might say.
Comments.UserDisplayName: quintumnia
Tags: <machine-learning><training><datasets>
Title: Train on big dataset (1mil + images)
UserDisplayName: Gaurav Gandhi
ThreadText: # Train on big dataset (1mil + images)
  
  (Asked by: username_0 on 2019-08-21)
  
  I am in the process of collecting a huge dataset of Human poses captured images to create a model to classify poses.
  
  My question is how will I be able to train on this massive dataset? I have multiple GPUs and Multiple machines access (Also have GCP).
  
  What would be the best way to train on such huge dataset?
  
  Thanks.
  
  **Tags:** machine-learning, training, datasets
  
  > **Comments:**
  > **username_1**: Welcome to Ai Community. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. Minimal, complete, verifiable example applies here. We cannot effectively help you until you post a sneak peak of  what you have tried out and accurately specify the problem.Sometimes,we leave out questions unanswered due to lack of a brief but descriptive summary of the problem as the title of the question might say.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-08-24. Score: 2 | Accepted answer
  
  is your data stored in raw ASCII text, like a CSV file?
  
  Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.
  
  There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.
  
  Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.
  
  In some cases, you may need to resort to a big data platform.
  That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.
  
  > **Comments:**
  > **username_0**: My data is in CSV having two columns, `img_path` and `label`. Image_path points to an actual image on which I have to train my CNN Model. So loading millions of images in memory is the main issue.  
  > **username_2**: how did u label your images ? did u use any tool or software ?.  
  
  ---
  
  