Answers.Body: <p>I suggest you <strong>fine-tune an existing model</strong>. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training. </p>
  
  <blockquote>
    <p>This way you will use few data to tune the model which leads to less
    memory use and less training time. You will also take advantage from a
    ready-to-use architecture and get accurate results.</p>
  </blockquote>
  
  <p>Depending on your case-study you can choose from this <em><a href="https://modelzoo.co/category/computer-vision" rel="nofollow noreferrer"><strong>list of computer vision pre-trained models.</strong></a></em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-17T15:30:26.963Z
Answers.LastActivityDate: 2019-05-17T15:30:26.963Z
Answers.UserDisplayName: HLeb
Answers.Body: <p>Most of the algorithms seem accessing the training set sequentially so the images need not be loaded into memory all at the same time.</p>
  
  <p>It is technically fully possible to build a workstation with 1 Tb of RAM or more, using a server barebone in a tower form factor (see <a href="https://www.supermicro.com/en/products/system/4U/7048/SYS-7048A-T.cfm?parts=SHOW" rel="nofollow noreferrer">this</a>, for instance, and would support multiple GPUs) but this only makes sense if the image loading is a bottleneck. Current SSDs are rather fast so you need to measure this before spending money on such a beast.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-02T07:05:47.313Z
Answers.LastActivityDate: 2019-07-02T07:05:47.313Z
Answers.UserDisplayName: h22
Body: <p>Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.</p>
  
  <p>From other research, I expect to be using about 200,000 training images at a resolution of 1280*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.</p>
  
  <p>I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.</p>
  
  <p>My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-31T15:10:00.060Z
Comments.Text: may I ask you, did you find out the solution? Like you said, if your training data is 550GB, how did you let CNN to train? Thank you so much.
Comments.UserDisplayName: Sunson29
Tags: <convolutional-neural-networks><datasets><memory>
Title: How to manage large amounts of image data for training?
UserDisplayName: Mitrimo
ThreadText: # How to manage large amounts of image data for training?
  
  (Asked by: username_0 on 2019-05-17)
  
  Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.
  
  From other research, I expect to be using about 200,000 training images at a resolution of 1280\*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.
  
  I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.
  
  My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?
  
  **Tags:** convolutional-neural-networks, datasets, memory
  > **Comments:**
  > **username_1**: may I ask you, did you find out the solution? Like you said, if your training data is 550GB, how did you let CNN to train? Thank you so much.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-05-17. Score: 1
  
  I suggest you **fine-tune an existing model**. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training.
  
  > This way you will use few data to tune the model which leads to less
  > memory use and less training time. You will also take advantage from a
  > ready-to-use architecture and get accurate results.
  
  Depending on your case-study you can choose from this *[**list of computer vision pre-trained models.**](https://modelzoo.co/category/computer-vision)*
  
  
  ---
  
  ### Answer by username_3 on 2019-07-02. Score: 1
  
  Most of the algorithms seem accessing the training set sequentially so the images need not be loaded into memory all at the same time.
  
  It is technically fully possible to build a workstation with 1 Tb of RAM or more, using a server barebone in a tower form factor (see [this](https://www.supermicro.com/en/products/system/4U/7048/SYS-7048A-T.cfm?parts=SHOW), for instance, and would support multiple GPUs) but this only makes sense if the image loading is a bottleneck. Current SSDs are rather fast so you need to measure this before spending money on such a beast.
  
  
  ---
  
  