Answers.Body: <p>Such large data cannot be loaded into your memory. Lets split what you can do into two:</p>
  
  <blockquote>
    <ol>
    <li><p>Rescale all your images to smaller dimensions. You can rescale them to 112x112 pixels. In your case, because you have a square image,
    there will be no need for cropping. You will still not be able to load
    all these images into your RAM at a goal.</p></li>
    <li><p>The best option is to use a generator function that will feed the data in batches. Please refer to the use of <a href="https://www.rdocumentation.org/packages/keras/versions/2.1.6/topics/fit_generator" rel="nofollow noreferrer">fit_generator</a> as used
    in Keras. If your model parameters become too big to fit into GPU
    memory, consider using batch normalization or using a Residual model
    to reduce your number of parameter.</p></li>
    </ol>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-28T15:22:54.863Z
Answers.Text: Why would you choose a size of 112x112 pixels? It is no potence of 2 and not a divisor of 2400.
Answers.UserDisplayName: Andy R
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-29T01:13:17.357Z
Answers.Text: @AndiR. Here is the thing. When it comes to input dimension, one is free to choose what ever size. This is because, if there are any dimension incompatibilities down in the network, this can easily be resolved using zero padding. Thus, there is no fixed methodology to the size of the input. What one must be careful about is making sure too much down sampling does  not affect the input quality.  Please take a look at this paper that makes use of the 112x112 dimension. (https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Tran_Learning_Spatiotemporal_Features_ICCV_2015_paper.pdf)
Answers.UserDisplayName: rocksyne
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-07T06:36:52.803Z
Answers.LastActivityDate: 2018-08-07T06:36:52.803Z
Answers.UserDisplayName: rocksyne
Answers.Body: <p>Usually for images the feature set is the pixel density values and in this case it will lead to quite a big feature set; also down sampling the images is also not recommended as you may lose (actually will) loose important data.</p>
  
  <p>[1] But there are some techniques that can help you reduce the feature set size, approaches like PCA(Principle Component Analysis) helps you in selection of important feature subset.</p>
  
  <p>For detailed information see link <a href="http://spark.apache.org/docs/latest/ml-features.html#pca" rel="noreferrer">http://spark.apache.org/docs/latest/ml-features.html#pca</a>.</p>
  
  <p>[2] Other than that to reduce the computational expense while training your Neural Network, you can use Stochastic Gradient Descent, rather than conventional use of Gradient Descent approach, that would reduce the size of dataset required for training in each iteration. Thus your dataset size to be used in one iteration would reduce, thus would reduce the time required to train the Network.</p>
  
  <p>The exact batch size to be used is dependent on your distribution for training dataset and testing datatset, a more general use is 70-30. Where you can also use above mentioned Stochastic approach to reduce required time.</p>
  
  <p>Detail for Stochastic Gradient Descent <a href="http://scikit-learn.org/stable/modules/sgd.html" rel="noreferrer">http://scikit-learn.org/stable/modules/sgd.html</a></p>
  
  <p>[3] The Hardware seems apt for the upgradation would be required, still if required look at cloud solutions like AWS where you can get free account subscription upto a limit of usage.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-01T09:03:45.067Z
Answers.LastActivityDate: 2017-09-01T09:03:45.067Z
Answers.UserDisplayName: Karan Chopra
Answers.Body: <blockquote>
    <p>How do I handle such large image sizes without downsampling?</p>
  </blockquote>
  
  <p>I assume that by downsampling you mean scaling down the input <em>before</em> passing it into CNN. <a href="http://cs231n.github.io/convolutional-networks/#conv" rel="noreferrer">Convolutional layer</a> allows to downsample the image within a network, by picking a large stride, which is going to save resources for the next layers. In fact, that's what it has to do, otherwise your model won't fit in GPU.</p>
  
  <blockquote>
    <ol>
    <li>Are there any techniques to handle such large images which are to be trained?</li>
    </ol>
  </blockquote>
  
  <p>Commonly researches scale the images to a resonable size. But if that's not an option for you, you'll need to restrict your CNN. In addition to downsampling in early layers, I would recommend you to get rid of FC layer (which normally takes most of parameters) <a href="http://cs231n.github.io/convolutional-networks/#convert" rel="noreferrer">in favor of convolutional layer</a>. Also you will have to stream your data in each epoch, because it won't fit into your GPU.</p>
  
  <p>Note that none of this will prevent heavy computational load in the early layers, exactly because the input is so large: convolution is an expensive operation and the first layers will perform <em>a lot</em> of them in each forward and backward pass. In short, training will be slow.</p>
  
  <blockquote>
    <ol start="2">
    <li>What batch size is reasonable to use?</li>
    </ol>
  </blockquote>
  
  <p>Here's another problem. A single image takes <code>2400x2400x3x4</code> (3 channels and 4 bytes per pixel) which is ~70Mb, so you can hardly afford even a batch size 10. More realistically would be 5. Note that most of the memory will be taken by CNN parameters. I think in this case it makes sense reduce the size by using 16-bit values rather than 32-bit - this way you'll be able to double the batches.</p>
  
  <blockquote>
    <ol start="3">
    <li>Are there any precautions to take, or any increase and decrease in hardware resources that I can do?</li>
    </ol>
  </blockquote>
  
  <p>Your bottleneck is GPU memory. If you can afford another GPU, get it and split the network across them. Everything else is insignificant compared to GPU memory.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-10-04T22:52:53.393Z
Answers.LastActivityDate: 2020-05-22T23:48:08.090Z
Answers.LastEditDate: 2020-05-22T23:48:08.090Z
Answers.UserDisplayName: Maxim
Body: <p>Suppose that I have 10K images of sizes <span class="math-container">$2400 \times 2400$</span> to train a CNN. </p>
  
  <p>How do I handle such large image sizes without downsampling?</p>
  
  <p>Here are a few more specific questions.</p>
  
  <ol>
  <li><p>Are there any techniques to handle such large images which are to be trained?</p></li>
  <li><p>What batch size is reasonable to use?</p></li>
  <li><p>Are there any precautions to take, or any increase and decrease in hardware resources that I can do?</p></li>
  </ol>
  
  <p>Here are the system requirements</p>
  
  <pre><code>Ubuntu 16.04 64-bit 
  RAM 16 GB
  GPU 8 GB
  HDD 500 GB
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-23T04:21:04.240Z
Comments.Text: I'd use conv layers with stride to downsample the input
Comments.UserDisplayName: Shimon Cohen
Tags: <neural-networks><deep-learning><image-recognition><convolutional-neural-networks>
Title: How do I handle large images when training a CNN?
UserDisplayName: WaterRocket8236
ThreadText: # How do I handle large images when training a CNN?
  
  (Asked by: username_0 on 2017-08-31)
  
  Suppose that I have 10K images of sizes $2400 \times 2400$ to train a CNN. 
  How do I handle such large image sizes without downsampling?
  Here are a few more specific questions.
  
  Are there any techniques to handle such large images which are to be trained?
  What batch size is reasonable to use?
  Are there any precautions to take, or any increase and decrease in hardware resources that I can do?
  
  Here are the system requirements
  Ubuntu 16.04 64-bit 
  RAM 16 GB
  GPU 8 GB
  HDD 500 GB
  
  **Tags:** neural-networks, deep-learning, image-recognition, convolutional-neural-networks
  > **Comments:**
  > **username_1**: I'd use conv layers with stride to downsample the input  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2017-10-04. Score: 23 | Accepted answer
  
  > How do I handle such large image sizes without downsampling?
  
  I assume that by downsampling you mean scaling down the input *before* passing it into CNN. [Convolutional layer](http://cs231n.github.io/convolutional-networks/#conv) allows to downsample the image within a network, by picking a large stride, which is going to save resources for the next layers. In fact, that's what it has to do, otherwise your model won't fit in GPU.
  
  > 1. Are there any techniques to handle such large images which are to be trained?
  
  Commonly researches scale the images to a resonable size. But if that's not an option for you, you'll need to restrict your CNN. In addition to downsampling in early layers, I would recommend you to get rid of FC layer (which normally takes most of parameters) [in favor of convolutional layer](http://cs231n.github.io/convolutional-networks/#convert). Also you will have to stream your data in each epoch, because it won't fit into your GPU.
  
  Note that none of this will prevent heavy computational load in the early layers, exactly because the input is so large: convolution is an expensive operation and the first layers will perform *a lot* of them in each forward and backward pass. In short, training will be slow.
  
  > 2. What batch size is reasonable to use?
  
  Here's another problem. A single image takes `2400x2400x3x4` (3 channels and 4 bytes per pixel) which is ~70Mb, so you can hardly afford even a batch size 10. More realistically would be 5. Note that most of the memory will be taken by CNN parameters. I think in this case it makes sense reduce the size by using 16-bit values rather than 32-bit - this way you'll be able to double the batches.
  
  > 3. Are there any precautions to take, or any increase and decrease in hardware resources that I can do?
  
  Your bottleneck is GPU memory. If you can afford another GPU, get it and split the network across them. Everything else is insignificant compared to GPU memory.
  
  
  ---
  
  ### Answer by username_4 on 2017-09-01. Score: 7
  
  Usually for images the feature set is the pixel density values and in this case it will lead to quite a big feature set; also down sampling the images is also not recommended as you may lose (actually will) loose important data.
  
  [1] But there are some techniques that can help you reduce the feature set size, approaches like PCA(Principle Component Analysis) helps you in selection of important feature subset.
  
  For detailed information see link <http://spark.apache.org/docs/latest/ml-features.html#pca>.
  
  [2] Other than that to reduce the computational expense while training your Neural Network, you can use Stochastic Gradient Descent, rather than conventional use of Gradient Descent approach, that would reduce the size of dataset required for training in each iteration. Thus your dataset size to be used in one iteration would reduce, thus would reduce the time required to train the Network.
  
  The exact batch size to be used is dependent on your distribution for training dataset and testing datatset, a more general use is 70-30. Where you can also use above mentioned Stochastic approach to reduce required time.
  
  Detail for Stochastic Gradient Descent <http://scikit-learn.org/stable/modules/sgd.html>
  
  [3] The Hardware seems apt for the upgradation would be required, still if required look at cloud solutions like AWS where you can get free account subscription upto a limit of usage.
  
  
  ---
  
  ### Answer by username_2 on 2018-08-07. Score: 3
  
  Such large data cannot be loaded into your memory. Lets split what you can do into two:
  
  > 1. Rescale all your images to smaller dimensions. You can rescale them to 112x112 pixels. In your case, because you have a square image,
  >    there will be no need for cropping. You will still not be able to load
  >    all these images into your RAM at a goal.
  > 2. The best option is to use a generator function that will feed the data in batches. Please refer to the use of [fit\_generator](https://www.rdocumentation.org/packages/keras/versions/2.1.6/topics/fit_generator) as used
  >    in Keras. If your model parameters become too big to fit into GPU
  >    memory, consider using batch normalization or using a Residual model
  >    to reduce your number of parameter.
  
  > **Comments:**
  > **username_3**: Why would you choose a size of 112x112 pixels? It is no potence of 2 and not a divisor of 2400.  
  > **username_2**: @AndiR. Here is the thing. When it comes to input dimension, one is free to choose what ever size. This is because, if there are any dimension incompatibilities down in the network, this can easily be resolved using zero padding. Thus, there is no fixed methodology to the size of the input. What one must be careful about is making sure too much down sampling does  not affect the input quality.  Please take a look at this paper that makes use of the 112x112 dimension. (https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Tran_Learning_Spatiotemporal_Features_ICCV_2015_paper.pdf)  
  
  ---
  
  