<p dir="auto">Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on <a href="http://stackoverflow.com/questions/tagged/keras" rel="nofollow">StackOverflow</a> or <a href="https://keras-slack-autojoin.herokuapp.com/" rel="nofollow">join the Keras Slack channel</a> and ask there instead of filing a GitHub issue.</p>
<p dir="auto">Thank you!</p>
<ul class="contains-task-list">
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Check that you are up-to-date with the master branch of Keras. You can update with:<br>
pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps</p>
</li>
<li class="task-list-item">
<p dir="auto"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found <a href="https://www.tensorflow.org/get_started/os_setup" rel="nofollow">here</a>.</p>
</li>
</ul>
<p dir="auto">I'm training a model on p2.8xlarge, my base model looks like this:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/10580847/36854173-ef76faa8-1d35-11e8-8399-08c35ff8e2b4.png"><img src="https://user-images.githubusercontent.com/10580847/36854173-ef76faa8-1d35-11e8-8399-08c35ff8e2b4.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto">And <code class="notranslate">parallel_model = multi_gpu_model(model, gpus=TOTAL_GPU)</code> looks like this,<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/10580847/36854093-c1c6fbb2-1d35-11e8-8194-af5cfd4642b8.png"><img src="https://user-images.githubusercontent.com/10580847/36854093-c1c6fbb2-1d35-11e8-8194-af5cfd4642b8.png" alt="image" style="max-width: 100%;"></a></p>
<p dir="auto"><strong>Warnings when initiating base model on CPU:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="WARNING:tensorflow:From /root/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
INFO:root:Model Instantiate on CPU"><pre class="notranslate"><code class="notranslate">WARNING:tensorflow:From /root/anaconda3/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
INFO:root:Model Instantiate on CPU
</code></pre></div>
<p dir="auto"><strong>Warnings when initiating parallel_model on GPU:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))"><pre class="notranslate"><code class="notranslate">/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
.
.
.
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
</code></pre></div>
<p dir="auto"><strong>Model training:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="TOTAL_GPU = 8
BATCH_SIZE = 512 * TOTAL_GPU
train_gen = generate_arrays_from_file(TRAIN_FEATURES, TRAIN_LABELS, BATCH_SIZE)
valid_gen = generate_arrays_from_file(VALID_FEATURES, VALID_LABELS, BATCH_SIZE)

train_data_count = 268200000
valid_data_count = 29800000

ST_PR_EP_TR = int(train_data_count // BATCH_SIZE)
ST_PR_EP_VA = int(valid_data_count // BATCH_SIZE)

history = parallel_model.fit_generator(
    train_gen,
    steps_per_epoch=ST_PR_EP_TR,
    epochs=100,
    verbose=0,
    validation_data=valid_gen,
    validation_steps=ST_PR_EP_VA,
    callbacks=callback_list,
    max_queue_size=80,
    use_multiprocessing=True,
    workers=8,
    shuffle=True,
    initial_epoch=0)"><pre class="notranslate"><code class="notranslate">TOTAL_GPU = 8
BATCH_SIZE = 512 * TOTAL_GPU
train_gen = generate_arrays_from_file(TRAIN_FEATURES, TRAIN_LABELS, BATCH_SIZE)
valid_gen = generate_arrays_from_file(VALID_FEATURES, VALID_LABELS, BATCH_SIZE)

train_data_count = 268200000
valid_data_count = 29800000

ST_PR_EP_TR = int(train_data_count // BATCH_SIZE)
ST_PR_EP_VA = int(valid_data_count // BATCH_SIZE)

history = parallel_model.fit_generator(
    train_gen,
    steps_per_epoch=ST_PR_EP_TR,
    epochs=100,
    verbose=0,
    validation_data=valid_gen,
    validation_steps=ST_PR_EP_VA,
    callbacks=callback_list,
    max_queue_size=80,
    use_multiprocessing=True,
    workers=8,
    shuffle=True,
    initial_epoch=0)
</code></pre></div>
<p dir="auto"><strong>Warnings when training model with fit_generator with batches:</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py:2088: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
  UserWarning('Using a generator with `use_multiprocessing=True`'"><pre class="notranslate"><code class="notranslate">WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_0/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:0 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_1/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:1 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_2/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:2 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_3/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:3 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_4/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:4 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_5/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:5 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_6/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:6 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/Shape with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
WARNING:tensorflow:Tried to colocate training/Adam/gradients/replica_7/sequential_1/embedding_1/Gather_grad/ToInt32 with an op embedding_1/embeddings/read that had a different device: /device:GPU:7 vs /device:CPU:0. Postponing error-checking until all devices are assigned.
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:45: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  if d.decorator_argspec is not None), _inspect.getargspec(target))
/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py:2088: UserWarning: Using a generator with `use_multiprocessing=True` and multiple workers may duplicate your data. Please consider using the`keras.utils.Sequence class.
  UserWarning('Using a generator with `use_multiprocessing=True`'
</code></pre></div>
<p dir="auto">Does <code class="notranslate">Postponing error-checking until all devices are assigned.</code> is somehow affecting model training ? or I can ignore this? I'm running Keras 2.1.4 and TensorFlow 1.6.0rc1 with Cuda compilation tools, release 9.0, V9.0.176</p>
<p dir="auto">Thanks!</p>
<p dir="auto">We should consider moving the modules <code class="notranslate">preprocessing</code> and <code class="notranslate">applications</code> to separate repositories under <code class="notranslate">keras-team</code>: <code class="notranslate">keras-team/preprocessing</code> and <code class="notranslate">keras-team/applications</code>.</p>
<p dir="auto">They would be listed as a dependency of <code class="notranslate">keras</code>, and would still be importable from e.g. <code class="notranslate">keras.preprocessing</code>.</p>
<p dir="auto">Why?</p>
<ul dir="auto">
<li>They're not part of "the core Keras API" (which is about model building and training)</li>
<li>They're not really coupled to the rest of the codebase, so unbundling is safe</li>
<li>Faster CI runs, better test coverage</li>
<li>Unbundling makes it easier to develop in an open-source setting</li>
<li>Easier to sync across <code class="notranslate">keras-team/keras</code> and <code class="notranslate">tf.keras</code> (shared dependencies instead of code duplication)</li>
<li>Other projects might have need of Numpy preprocessing utilities</li>
</ul>
<p dir="auto">Any comments or concerns? Who would be interested in taking greater ownership of these repos, should they materialize?</p>