<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Custom</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Windows 10</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Binary</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.9-rc0</li>
<li><strong>Python version</strong>: 3.6</li>
<li><strong>Bazel version (if compiling from source)</strong>: NA</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: NA</li>
<li><strong>CUDA/cuDNN version</strong>: NA</li>
<li><strong>GPU model and memory</strong>: NA</li>
<li><strong>Exact command to reproduce</strong>: <code class="notranslate">model.fit(get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)</code><br>
and<br>
<code class="notranslate">model.fit(get_iterator,get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)</code></li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">When I pass one dataset iterator to <code class="notranslate">fit</code> method, I get:</p>
<blockquote>
<p dir="auto">Please provide data as a list or tuple of 2 elements  - input and target pair. Received Tensor("IteratorGetNext_4:0", shape=(2, ?), dtype=float32)</p>
</blockquote>
<p dir="auto">When I pass two iterators I get the error:</p>
<blockquote>
<p dir="auto">ValueError: You passed a dataset or dataset iterator (&lt;tensorflow.python.data.ops.iterator_ops.Iterator object at 0x000001FEABE88748&gt;) as input <code class="notranslate">x</code> to your model. In that case, you should not specify a target (<code class="notranslate">y</code>) argument, since the dataset or dataset iterator generates both input data and target data. Received: &lt;tensorflow.python.data.ops.iterator_ops.Iterator object at 0x000001FEABE88748&gt;</p>
</blockquote>
<p dir="auto">When I create a new dataset after zipping the original x and y data set and pass that to <code class="notranslate">fit </code>I get the error described in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="331304024" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/19912" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/19912/hovercard" href="https://github.com/tensorflow/tensorflow/issues/19912">#19912</a></p>
<p dir="auto">According to 1.9-rc0 method release notes iterators should be usable with keras training methods. Please provide a solution or provide clarification in the documentation.</p>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="dataset= tf.contrib.data.make_csv_dataset(file_name,48,select_columns= ['Load_residential_multi_0','Load_residential_multi_1'],shuffle=False)
dataset = dataset.map(lambda x: tf.stack(list(x.values())))
get_iterator = dataset.make_one_shot_iterator()
get_batch = get_iterator.get_next()

#Building and training a single layer model using Keras (Available within TensorFlow)
model = Sequential() 
#Input Layer
model.add(InputLayer(input_shape=(48,),name='InputLayer'))#,input_tensor =dataset
#model.add(BatchNormalization(axis=1))  #Normalizing values
#Layer1 
model.add(Dense(units=5,activation='relu',name='FeedForward1'))  #Add a feed forward layer
#Layer2 
model.add(Dense(units=5,activation='relu',name='FeedForward2'))  #Add a feed forward layer
#Output layer 
model.add(Dense(units=48,name='OutputLayer'))

#Specify los function and optimizer
model.compile(loss='mse',optimizer='adam',metrics=['mae'])

#Summarize model
model.summary()
#Train the model
model.fit(get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)
#model.fit(get_iterator,get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)"><pre class="notranslate"><code class="notranslate">dataset= tf.contrib.data.make_csv_dataset(file_name,48,select_columns= ['Load_residential_multi_0','Load_residential_multi_1'],shuffle=False)
dataset = dataset.map(lambda x: tf.stack(list(x.values())))
get_iterator = dataset.make_one_shot_iterator()
get_batch = get_iterator.get_next()

#Building and training a single layer model using Keras (Available within TensorFlow)
model = Sequential() 
#Input Layer
model.add(InputLayer(input_shape=(48,),name='InputLayer'))#,input_tensor =dataset
#model.add(BatchNormalization(axis=1))  #Normalizing values
#Layer1 
model.add(Dense(units=5,activation='relu',name='FeedForward1'))  #Add a feed forward layer
#Layer2 
model.add(Dense(units=5,activation='relu',name='FeedForward2'))  #Add a feed forward layer
#Output layer 
model.add(Dense(units=48,name='OutputLayer'))

#Specify los function and optimizer
model.compile(loss='mse',optimizer='adam',metrics=['mae'])

#Summarize model
model.summary()
#Train the model
model.fit(get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)
#model.fit(get_iterator,get_iterator,steps_per_epoch=2,batch_size=2,epochs=2,shuffle =True,verbose=1)
</code></pre></div>
<p dir="auto">Hi,</p>
<p dir="auto">I ran into this performance issue while trying to upgrade tensorflow from version 0.12.1  to 1.X.</p>
<p dir="auto">We ran a network with large embedding lookup tables:</p>
<ul dir="auto">
<li>100K X 32 (for example, word embedding -  with 100K unique words)</li>
<li>300K X 128 (for example, categorical feature with cardinality of 300K unique items)</li>
</ul>
<p dir="auto">After upgrading TF version to 1.0.1,  GPU usage dropped in from 60% to 30%.<br>
Training time went up in 50%-200% (depends on how big is the embedding lookup table).</p>
<p dir="auto">This is the commit that caused the performance degradation:<br>
<a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/tensorflow/tensorflow/commit/f9f56f9dc7fe41ef1128290a77ac88e889ea5229/hovercard" href="https://github.com/tensorflow/tensorflow/commit/f9f56f9dc7fe41ef1128290a77ac88e889ea5229"><tt>f9f56f9</tt></a></p>
<p dir="auto">The handling of unique indexes is very slow and does not run in parallel with others operations.<br>
Please note the big unique blocks in the middle.<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/8734262/26542969/ab0f3740-4464-11e7-9dcb-f3ccd58dfc8a.png"><img src="https://cloud.githubusercontent.com/assets/8734262/26542969/ab0f3740-4464-11e7-9dcb-f3ccd58dfc8a.png" alt="trace_unique" style="max-width: 100%;"></a></p>
<p dir="auto">Here is a work around (not handling unique indexes ):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class MyOptimizer(tf.train.AdamOptimizer):
        def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8,
               use_locking=False, name=&quot;Adam&quot;):
                super(MyOptimizer,self).__init__(learning_rate,beta1, beta2, epsilon, use_locking,name)

        def _apply_sparse_duplicate_indices(self, grad, var):
                return self._apply_sparse(grad, var)"><pre class="notranslate"><code class="notranslate">class MyOptimizer(tf.train.AdamOptimizer):
        def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8,
               use_locking=False, name="Adam"):
                super(MyOptimizer,self).__init__(learning_rate,beta1, beta2, epsilon, use_locking,name)

        def _apply_sparse_duplicate_indices(self, grad, var):
                return self._apply_sparse(grad, var)
</code></pre></div>
<p dir="auto">Thanks,<br>
Erez</p>