<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No</li>
<li>TensorFlow installed from (source or binary): binary</li>
<li>TensorFlow version (use command below): v2.0.0-beta1-5101-gc75bb66 2.0.0-rc0</li>
<li>Python version: 3</li>
<li>Bazel version (if compiling from source): No</li>
<li>GCC/Compiler version (if compiling from source): No</li>
<li>CUDA/cuDNN version: Unknown, from Colab</li>
<li>GPU model and memory: Unknown, from Colab</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong><br>
When i rewrote my estimator-based model (Dataset + Feature columns) to keras i was able to run (train) it.</p>
<p dir="auto">But when i converted it to estimator as shown in migration guide <a href="https://www.tensorflow.org/beta/guide/migration_guide#estimators" rel="nofollow">https://www.tensorflow.org/beta/guide/migration_guide#estimators</a> it fails with error</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-7-6538390f0054&gt; in &lt;module&gt;()
      1 estimator = tf.keras.estimator.model_to_estimator(
----&gt; 2   keras_model = model
      3 )

9 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--&gt; 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:
    relative to /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/feature_column:

    sequence_feature_column.py:140 call
        transformation_cache, self._state_manager)
    feature_column_v2.py:3205 get_sequence_dense_tensor
        self.categorical_column))

    ValueError: In embedding_column: words_embedding. categorical_column must be of type SequenceCategoricalColumn to use SequenceFeatures. Suggested fix: Use one of sequence_categorical_column_with_*. Given (type &lt;class 'tensorflow.python.feature_column.feature_column_v2.HashedCategoricalColumn'&gt;): HashedCategoricalColumn(key='words', hash_bucket_size=1000, dtype=tf.string)"><pre class="notranslate"><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span>
<span class="pl-v">ValueError</span>                                <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>)
<span class="pl-c1">&lt;</span><span class="pl-s1">ipython</span><span class="pl-c1">-</span><span class="pl-s1">input</span><span class="pl-c1">-</span><span class="pl-c1">7</span><span class="pl-c1">-</span><span class="pl-c1">6538390</span><span class="pl-s1">f0054</span><span class="pl-c1">&gt;</span> <span class="pl-s1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>()
      <span class="pl-c1">1</span> <span class="pl-s1">estimator</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">keras</span>.<span class="pl-s1">estimator</span>.<span class="pl-en">model_to_estimator</span>(
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">2</span>   <span class="pl-s1">keras_model</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>
      <span class="pl-c1">3</span> )

<span class="pl-c1">9</span> <span class="pl-s1">frames</span>
<span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow_core</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">autograph</span><span class="pl-c1">/</span><span class="pl-s1">impl</span><span class="pl-c1">/</span><span class="pl-s1">api</span>.<span class="pl-s1">py</span> <span class="pl-c1">in</span> <span class="pl-en">wrapper</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
    <span class="pl-c1">235</span>       <span class="pl-s1">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:  <span class="pl-c"># pylint:disable=broad-except</span>
    <span class="pl-c1">236</span>         <span class="pl-k">if</span> <span class="pl-en">hasattr</span>(<span class="pl-s1">e</span>, <span class="pl-s">'ag_error_metadata'</span>):
<span class="pl-c1">-</span><span class="pl-c1">-</span><span class="pl-c1">&gt;</span> <span class="pl-c1">237</span>           <span class="pl-s1">raise</span> <span class="pl-s1">e</span>.<span class="pl-s1">ag_error_metadata</span>.<span class="pl-en">to_exception</span>(<span class="pl-s1">e</span>)
    <span class="pl-c1">238</span>         <span class="pl-k">else</span>:
    <span class="pl-c1">239</span>           <span class="pl-s1">raise</span>

<span class="pl-v">ValueError</span>: <span class="pl-s1">in</span> <span class="pl-s1">converted</span> <span class="pl-s1">code</span>:
    <span class="pl-s1">relative</span> <span class="pl-s1">to</span> <span class="pl-c1">/</span><span class="pl-s1">usr</span><span class="pl-c1">/</span><span class="pl-s1">local</span><span class="pl-c1">/</span><span class="pl-s1">lib</span><span class="pl-c1">/</span><span class="pl-s1">python3</span>.<span class="pl-c1">6</span><span class="pl-c1">/</span><span class="pl-s1">dist</span><span class="pl-c1">-</span><span class="pl-s1">packages</span><span class="pl-c1">/</span><span class="pl-s1">tensorflow_core</span><span class="pl-c1">/</span><span class="pl-s1">python</span><span class="pl-c1">/</span><span class="pl-s1">feature_column</span>:

    <span class="pl-s1">sequence_feature_column</span>.<span class="pl-s1">py</span>:<span class="pl-c1">140</span> <span class="pl-s1">call</span>
        <span class="pl-s1">transformation_cache</span>, <span class="pl-s1">self</span>.<span class="pl-s1">_state_manager</span>)
    <span class="pl-s1">feature_column_v2</span>.<span class="pl-s1">py</span>:<span class="pl-c1">3205</span> <span class="pl-s1">get_sequence_dense_tensor</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">categorical_column</span>))

    <span class="pl-v">ValueError</span>: <span class="pl-v">In</span> <span class="pl-s1">embedding_column</span>: <span class="pl-s1">words_embedding</span>. <span class="pl-s1">categorical_column</span> <span class="pl-s1">must</span> <span class="pl-s1">be</span> <span class="pl-s1">of</span> <span class="pl-s1">type</span> <span class="pl-v">SequenceCategoricalColumn</span> <span class="pl-s1">to</span> <span class="pl-s1">use</span> <span class="pl-v">SequenceFeatures</span>. <span class="pl-v">Suggested</span> <span class="pl-s1">fix</span>: <span class="pl-v">Use</span> <span class="pl-s1">one</span> <span class="pl-s1">of</span> <span class="pl-s1">sequence_categorical_column_with_</span><span class="pl-c1">*</span>. <span class="pl-v">Given</span> (<span class="pl-s1">type</span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'tensorflow.python.feature_column.feature_column_v2.HashedCategoricalColumn'</span><span class="pl-c1">&gt;</span>): <span class="pl-v">HashedCategoricalColumn</span>(<span class="pl-s1">key</span><span class="pl-c1">=</span><span class="pl-s">'words'</span>, <span class="pl-s1">hash_bucket_size</span><span class="pl-c1">=</span><span class="pl-c1">1000</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">string</span>)</pre></div>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
Working keras model should always be convertable to estimator.</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Here is an example <a href="https://colab.research.google.com/drive/11pWyltRzvQPPvM2dWQ8EqxafB9ByCxNn" rel="nofollow">https://colab.research.google.com/drive/11pWyltRzvQPPvM2dWQ8EqxafB9ByCxNn</a><br>
There are 2 cases shown. First i show that keras model with sequence features is working. See <code class="notranslate">model.fit_generator</code>.<br>
And then i show that conversion to estimator fails.</p>
<hr>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04</li>
<li><strong>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device</strong>: N/A</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Source</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.8</li>
<li><strong>Python version</strong>: 2.7</li>
<li><strong>Bazel version (if compiling from source)</strong>: 0.11.1</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: gcc4.8</li>
<li><strong>CUDA/cuDNN version</strong>: N/A</li>
<li><strong>GPU model and memory</strong>: AMD R9 Nano</li>
<li><strong>Exact command to reproduce</strong>: N/A</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.</p>
<p dir="auto">Hi,<br>
We are working on a SYCL implementation of TensorFlow. When implementing the backend for the scatter_nd operation we were wondering what is the behavior of concurrent updates.<br>
The test in <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/kernel_tests/scatter_nd_ops_test.py#L231">scatter_nd_ops_test.py</a> assumes that the operations are applied in a non-deterministic order, which matches what the <a href="https://www.tensorflow.org/api_docs/python/tf/scatter_nd" rel="nofollow">documentation</a> says. This corner case is costly to handle on the GPU and it seems that the <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/scatter_nd_op_gpu.cu.cc#L81">CUDA</a> implementation does not handle it, neither does ours.</p>
<p dir="auto">The documentation says:</p>
<blockquote>
<p dir="auto">The order in which updates are applied is nondeterministic, so the output will be nondeterministic if indices contains duplicates.</p>
</blockquote>
<p dir="auto">The test checks for Add and Sub of two concurrent indices to result in the total summation (or subtraction) of both updates to apply (order of additions is nondeterministic but the result is). On a replace operation the output would be nondeterministic (hence the test not checking for it).<br>
With a non blocking GPU implementation the second update applied to an index may (and will probably) overwrite the first update from a different thread.</p>
<p dir="auto">Our question then is:<br>
Is the test wrong to test that concurrent summation &amp; subtraction is deterministic (in which case we would be happy to write a PR to correct that)?<br>
Otherwise, is the CUDA implementation wrong? In which case the documentation should probably be more descriptive on what happens with concurrent add/sub compared to replace.</p>
<p dir="auto">Thanks,</p>