title: Tensorflow Out of memory and CPU/GPU usage
body: <p>I am using Tensorflow with Keras to train a neural network for object recognition (YOLO).</p>
  
  <p>I wrote the model and I am trying to train it using keras model.fit_generator() with batches of 32 416x416x3 images.</p>
  
  <p>I am using a NVIDIA GEFORCE RTX 2070 GPU with 8GB memory (Tensorflow uses about 6.6 GB).</p>
  
  <p>However when I start training the model I receive messages like this:</p>
  
  <pre><code>tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape
  
  2019-02-11 16:13:08.051289: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 338.00MiB.  Current allocation summary follows.
  2019-02-11 16:13:08.057318: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 1589, Chunks in use: 1589. 397.3KiB allocated for chunks. 397.3KiB in use in bin. 25.2KiB client-requested in use in bin.
  2019-02-11 16:13:08.061222: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 204, Chunks in use: 204. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 100.1KiB client-requested in use in bin.
  ...
  2019-02-11 16:13:08.142674: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 11, Chunks in use: 11. 5.05GiB allocated for chunks. 5.05GiB in use in bin. 4.95GiB client-requested in use in bin.
  2019-02-11 16:13:08.148149: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 338.00MiB was 256.00MiB, Chunk State:
  2019-02-11 16:13:08.150474: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B400000 of size 1280
  2019-02-11 16:13:08.152627: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B400500 of size 256
  2019-02-11 16:13:08.154790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 000000070B400600 of size 256
  ....
  2019-02-11 16:17:38.699526: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 6.11GiB
  2019-02-11 16:17:38.701621: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
  Limit:                  6624727531
  InUse:                  6557567488
  MaxInUse:               6590199040
  NumAllocs:                    3719
  MaxAllocSize:           1624768512
  
  2019-02-11 16:17:38.708981: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ****************************************************************************************************
  2019-02-11 16:17:38.712172: W tensorflow/core/framework/op_kernel.cc:1412] OP_REQUIRES failed at conv_ops_fused.cc:734 : Resource exhausted: OOM when allocating tensor with shape[16,256,52,52] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
  </code></pre>
  
  <p>I reported only few lines of that message but it seems clear that is a memory usage problem.</p>
  
  <p>Maybe should I use CPU in my generator function for reading images and labels from files?
  In this case how should I do?</p>
  
  <p>Thank you.</p>