<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>:<br>
Yes, script is below</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>:<br>
Linux 4.15.0-23-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="115938248" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/25" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/25/hovercard" href="https://github.com/tensorflow/tensorflow/issues/25">#25</a>-Ubuntu SMP Wed May 23 18:02:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux<br>
VERSION="18.04 LTS (Bionic Beaver)"</li>
<li><strong>TensorFlow installed from (source or binary)</strong>:<br>
binary</li>
<li><strong>TensorFlow version (use command below)</strong>:<br>
v1.8.0-0-g93bc2e2072 1.8.0</li>
<li><strong>Python version</strong>:<br>
Python 3.6.5</li>
<li><strong>Bazel version (if compiling from source)</strong>: n/a</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: n/a</li>
<li><strong>CUDA/cuDNN version</strong>: n/a, using CPU version</li>
<li><strong>GPU model and memory</strong>: n/a using CPU</li>
<li><strong>Exact command to reproduce</strong>: just run "python3 script.py"</li>
</ul>
<h3 dir="auto">Describe the problem</h3>
<p dir="auto">When using the tf.where function where a division by zero exists in one of the two where branches, you get a NaN gradient even if the division by zero was on the where branch which was not executed.</p>
<p dir="auto">This seems similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="157280822" data-permission-text="Title is private" data-url="https://github.com/tensorflow/tensorflow/issues/2540" data-hovercard-type="issue" data-hovercard-url="/tensorflow/tensorflow/issues/2540/hovercard" href="https://github.com/tensorflow/tensorflow/issues/2540">#2540</a> but the workarounds suggested there (e.g. using tf.boolean_mask) did not work.</p>
<h3 dir="auto">Source code / logs</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow as tf
sess = tf.Session()
W1 = tf.Variable([2.0])
W2 = tf.Variable([0.0])
output=tf.where(W1&gt;4, W1/W2, tf.zeros_like(W1))  # gives correct answer (zero) since W1&gt;4 is false
gradient=tf.gradients(output, W2)[0] # should be zero, but it gives NaN
sess.run(tf.global_variables_initializer())
print(sess.run([output, gradient]))"><pre class="notranslate"><code class="notranslate">import tensorflow as tf
sess = tf.Session()
W1 = tf.Variable([2.0])
W2 = tf.Variable([0.0])
output=tf.where(W1&gt;4, W1/W2, tf.zeros_like(W1))  # gives correct answer (zero) since W1&gt;4 is false
gradient=tf.gradients(output, W2)[0] # should be zero, but it gives NaN
sess.run(tf.global_variables_initializer())
print(sess.run([output, gradient]))
</code></pre></div>
<h1 dir="auto">Program output:</h1>
<p dir="auto">#[array([0.], dtype=float32), array([nan], dtype=float32)]</p>
<p dir="auto">Hello!</p>
<p dir="auto">I've recently updated to tensorflow version 1.5.0, and suddenly receive an error, that I can't decipher, for code that worked before (in version 1.4.1): <code class="notranslate">Cannot use 'transducer_training/while/rnn/strided_slice' as input to 'gradients/transducer_training/while/rnn/while/Select_1_grad/Select/f_acc' because 'transducer_training/while/rnn/strided_slice' is in a while loop</code></p>
<p dir="auto">I've also tried using the softmax_cross_entropy_with_logits function, but that still produced the same error. Here's the <a href="https://stackoverflow.com/questions/48713335/tensorflow-strided-slice-slicing-error-with-while-loop" rel="nofollow">stackoverflow</a> post, in case its a coding mistake on my part.<br>
The model is a seq2seq variation.</p>
<hr>
<h3 dir="auto">System information</h3>
<ul dir="auto">
<li><strong>Have I written custom code (as opposed to using a stock example script provided in TensorFlow)</strong>: Yes</li>
<li><strong>OS Platform and Distribution (e.g., Linux Ubuntu 16.04)</strong>: Ubuntu 16.04</li>
<li><strong>TensorFlow installed from (source or binary)</strong>: Binary</li>
<li><strong>TensorFlow version (use command below)</strong>: 1.5.0 (previous 1.4.1)</li>
<li><strong>Python version</strong>: 2.7</li>
<li><strong>Bazel version (if compiling from source)</strong>: N/A</li>
<li><strong>GCC/Compiler version (if compiling from source)</strong>: N/A</li>
<li><strong>CUDA/cuDNN version</strong>: N/A</li>
<li><strong>GPU model and memory</strong>: N/A</li>
<li><strong>Exact command to reproduce</strong>: Copy, paste and run the code</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import tensorflow as tf
from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple
from tensorflow.python.layers import core as layers_core

# NOTE: Time major

# ---------------- Constants Manager ----------------------------
class ConstantsManager(object):
    def __init__(self, input_dimensions, input_embedding_size, inputs_embedded, encoder_hidden_units,
                 transducer_hidden_units, vocab_ids, input_block_size, beam_width):
        assert transducer_hidden_units == encoder_hidden_units, 'Encoder and transducer have to have the same amount' \
                                                                'of hidden units'
        self.input_dimensions = input_dimensions
        self.vocab_ids = vocab_ids
        self.E_SYMBOL = len(self.vocab_ids)
        self.vocab_ids.append('E_SYMBOL')
        self.GO_SYMBOL = len(self.vocab_ids)
        self.vocab_ids.append('GO_SYMBOL')
        self.vocab_size = len(self.vocab_ids)
        self.input_embedding_size = input_embedding_size
        self.inputs_embedded = inputs_embedded
        self.encoder_hidden_units = encoder_hidden_units
        self.transducer_hidden_units = transducer_hidden_units
        self.input_block_size = input_block_size
        self.beam_width = beam_width
        self.batch_size = 1  # Cannot be increased, see paper
        self.log_prob_init_value = 0

# ----------------- Model ---------------------------------------


class Model(object):
    def __init__(self, cons_manager):
        self.var_list = []
        self.cons_manager = cons_manager
        self.max_blocks, self.inputs_full_raw, self.transducer_list_outputs, self.start_block, self.encoder_hidden_init,\
            self.trans_hidden_init, self.logits, self.encoder_hidden_state_new, \
            self.transducer_hidden_state_new, self.train_saver = self.build_full_transducer()

        self.targets, self.train_op, self.loss = self.build_training_step()

    def build_full_transducer(self):
        with tf.variable_scope('transducer_training'):

            embeddings = tf.Variable(tf.random_uniform([self.cons_manager.vocab_size,
                                                        self.cons_manager.input_embedding_size], -1.0, 1.0),
                                     dtype=tf.float32,
                                     name='embedding')
            # Inputs
            max_blocks = tf.placeholder(dtype=tf.int32, name='max_blocks')  # total amount of blocks to go through
            if self.cons_manager.inputs_embedded is True:
                input_type = tf.float32
            else:
                input_type = tf.int32
            inputs_full_raw = tf.placeholder(shape=(None, self.cons_manager.batch_size,
                                                    self.cons_manager.input_dimensions), dtype=input_type,
                                             name='inputs_full_raw')  # shape [max_time, 1, input_dims]
            transducer_list_outputs = tf.placeholder(shape=(None,), dtype=tf.int32,
                                                     name='transducer_list_outputs')  # amount to output per block
            start_block = tf.placeholder(dtype=tf.int32, name='transducer_start_block')  # where to start the input

            encoder_hidden_init = tf.placeholder(shape=(2, 1, self.cons_manager.encoder_hidden_units), dtype=tf.float32,
                                                 name='encoder_hidden_init')
            trans_hidden_init = tf.placeholder(shape=(2, 1, self.cons_manager.transducer_hidden_units), dtype=tf.float32,
                                               name='trans_hidden_init')

            # Temporary constants, maybe changed during inference
            end_symbol = tf.get_variable(name='end_symbol',
                                         initializer=tf.constant_initializer(self.cons_manager.vocab_size),
                                         shape=(), dtype=tf.int32)

            # Turn inputs into tensor which is easily readable#

            inputs_full = tf.reshape(inputs_full_raw, shape=[-1, self.cons_manager.input_block_size,
                                                             self.cons_manager.batch_size,
                                                             self.cons_manager.input_dimensions])

            # Outputs
            outputs_ta = tf.TensorArray(dtype=tf.float32, size=max_blocks)

            init_state = (start_block, outputs_ta, encoder_hidden_init, trans_hidden_init)

            # Initiate cells, NOTE: if there is a future error, put these back inside the body function
            encoder_cell = tf.contrib.rnn.LSTMCell(num_units=self.cons_manager.encoder_hidden_units)
            transducer_cell = tf.contrib.rnn.LSTMCell(self.cons_manager.transducer_hidden_units)

            def cond(current_block, outputs_int, encoder_hidden, trans_hidden):
                return current_block &lt; start_block + max_blocks

            def body(current_block, outputs_int, encoder_hidden, trans_hidden):

                # --------------------- ENCODER ----------------------------------------------------------------------
                encoder_inputs = inputs_full[current_block]
                encoder_inputs_length = [tf.shape(encoder_inputs)[0]]
                encoder_hidden_state = encoder_hidden

                if self.cons_manager.inputs_embedded is True:
                    encoder_inputs_embedded = encoder_inputs
                else:
                    encoder_inputs = tf.reshape(encoder_inputs, shape=[-1, self.cons_manager.batch_size])
                    encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)

                # Build model

                # Build previous state
                encoder_hidden_c, encoder_hidden_h = tf.split(encoder_hidden_state, num_or_size_splits=2, axis=0)
                encoder_hidden_c = tf.reshape(encoder_hidden_c, shape=[-1, self.cons_manager.encoder_hidden_units])
                encoder_hidden_h = tf.reshape(encoder_hidden_h, shape=[-1, self.cons_manager.encoder_hidden_units])
                encoder_hidden_state_t = LSTMStateTuple(encoder_hidden_c, encoder_hidden_h)

                #   encoder_outputs: [max_time, batch_size, num_units]
                encoder_outputs, encoder_hidden_state_new = tf.nn.dynamic_rnn(
                    encoder_cell, encoder_inputs_embedded,
                    sequence_length=encoder_inputs_length, time_major=True,
                    dtype=tf.float32, initial_state=encoder_hidden_state_t)

                # Modify output of encoder_hidden_state_new so that it can be fed back in again without problems.
                encoder_hidden_state_new = tf.concat([encoder_hidden_state_new.c, encoder_hidden_state_new.h], axis=0)
                encoder_hidden_state_new = tf.reshape(encoder_hidden_state_new,
                                                      shape=[2, -1, self.cons_manager.encoder_hidden_units])

                # --------------------- TRANSDUCER --------------------------------------------------------------------
                encoder_raw_outputs = encoder_outputs
                # Save/load the state as one tensor, use encoder state as init if this is the first block
                trans_hidden_state = tf.cond(current_block &gt; 0, lambda: trans_hidden, lambda: encoder_hidden_state_new)
                transducer_amount_outputs = transducer_list_outputs[current_block - start_block]

                # Model building
                helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(
                    embedding=embeddings,
                    start_tokens=tf.tile([self.cons_manager.GO_SYMBOL],
                                         [self.cons_manager.batch_size]),  # TODO: check if this looks good
                    end_token=end_symbol)  # vocab size, so that it doesn't prematurely end the decoding

                attention_states = tf.transpose(encoder_raw_outputs,
                                                [1, 0, 2])  # attention_states: [batch_size, max_time, num_units]

                attention_mechanism = tf.contrib.seq2seq.LuongAttention(
                    self.cons_manager.encoder_hidden_units, attention_states)

                decoder_cell = tf.contrib.seq2seq.AttentionWrapper(
                    transducer_cell,
                    attention_mechanism,
                    attention_layer_size=self.cons_manager.transducer_hidden_units)

                projection_layer = layers_core.Dense(self.cons_manager.vocab_size, use_bias=False)

                # Build previous state
                trans_hidden_c, trans_hidden_h = tf.split(trans_hidden_state, num_or_size_splits=2, axis=0)
                trans_hidden_c = tf.reshape(trans_hidden_c, shape=[-1, self.cons_manager.transducer_hidden_units])
                trans_hidden_h = tf.reshape(trans_hidden_h, shape=[-1, self.cons_manager.transducer_hidden_units])
                trans_hidden_state_t = LSTMStateTuple(trans_hidden_c, trans_hidden_h)

                decoder = tf.contrib.seq2seq.BasicDecoder(
                    decoder_cell, helper,
                    decoder_cell.zero_state(1, tf.float32).clone(cell_state=trans_hidden_state_t),
                    output_layer=projection_layer)

                outputs, transducer_hidden_state_new, _ = tf.contrib.seq2seq.dynamic_decode(decoder,
                                                                                            output_time_major=True,
                                                                                            maximum_iterations=transducer_amount_outputs)
                logits = outputs.rnn_output  # logits of shape [max_time,batch_size,vocab_size]
                decoder_prediction = outputs.sample_id  # For debugging

                # Modify output of transducer_hidden_state_new so that it can be fed back in again without problems.
                transducer_hidden_state_new = tf.concat(
                    [transducer_hidden_state_new[0].c, transducer_hidden_state_new[0].h],
                    axis=0)
                transducer_hidden_state_new = tf.reshape(transducer_hidden_state_new,
                                                         shape=[2, -1, self.cons_manager.transducer_hidden_units])


                # Note the outputs
                outputs_int = outputs_int.write(current_block - start_block, logits)

                return current_block + 1, outputs_int, encoder_hidden_state_new, transducer_hidden_state_new

            _, outputs_final, encoder_hidden_state_new, transducer_hidden_state_new = \
                tf.while_loop(cond, body, init_state, parallel_iterations=1)

            # Process outputs
            outputs = outputs_final.concat()
            logits = tf.reshape(
                outputs,
                shape=(-1, 1, self.cons_manager.vocab_size))  # And now its [max_output_time, batch_size, vocab]

            # For loading the model later on
            logits = tf.identity(logits, name='logits')
            encoder_hidden_state_new = tf.identity(encoder_hidden_state_new, name='encoder_hidden_state_new')
            transducer_hidden_state_new = tf.identity(transducer_hidden_state_new, name='transducer_hidden_state_new')

        train_saver = tf.train.Saver()  # For now save everything

        return max_blocks, inputs_full_raw, transducer_list_outputs, start_block, encoder_hidden_init,\
            trans_hidden_init, logits, encoder_hidden_state_new, transducer_hidden_state_new, train_saver

    def build_training_step(self):
        targets = tf.placeholder(shape=(None,), dtype=tf.int32, name='targets')
        targets_one_hot = tf.one_hot(targets, depth=self.cons_manager.vocab_size, dtype=tf.float32)

        targets_one_hot = tf.Print(targets_one_hot, [targets], message='Targets: ', summarize=10)
        targets_one_hot = tf.Print(targets_one_hot, [tf.argmax(self.logits, axis=2)], message='Argmax: ', summarize=10)

        stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=targets_one_hot,
                                                                         logits=self.logits)
        loss = tf.reduce_mean(stepwise_cross_entropy)
        train_op = tf.train.AdamOptimizer().minimize(loss)
        return targets, train_op, loss


constants_manager = ConstantsManager(input_dimensions=1, input_embedding_size=11, inputs_embedded=False,
                                     encoder_hidden_units=100, transducer_hidden_units=100, vocab_ids=[0, 1, 2],
                                     input_block_size=1, beam_width=5)
model = Model(cons_manager=constants_manager)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">tensorflow</span> <span class="pl-k">as</span> <span class="pl-s1">tf</span>
<span class="pl-k">from</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">rnn</span> <span class="pl-k">import</span> <span class="pl-v">LSTMCell</span>, <span class="pl-v">LSTMStateTuple</span>
<span class="pl-k">from</span> <span class="pl-s1">tensorflow</span>.<span class="pl-s1">python</span>.<span class="pl-s1">layers</span> <span class="pl-k">import</span> <span class="pl-s1">core</span> <span class="pl-k">as</span> <span class="pl-s1">layers_core</span>

<span class="pl-c"># NOTE: Time major</span>

<span class="pl-c"># ---------------- Constants Manager ----------------------------</span>
<span class="pl-k">class</span> <span class="pl-v">ConstantsManager</span>(<span class="pl-s1">object</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">input_dimensions</span>, <span class="pl-s1">input_embedding_size</span>, <span class="pl-s1">inputs_embedded</span>, <span class="pl-s1">encoder_hidden_units</span>,
                 <span class="pl-s1">transducer_hidden_units</span>, <span class="pl-s1">vocab_ids</span>, <span class="pl-s1">input_block_size</span>, <span class="pl-s1">beam_width</span>):
        <span class="pl-k">assert</span> <span class="pl-s1">transducer_hidden_units</span> <span class="pl-c1">==</span> <span class="pl-s1">encoder_hidden_units</span>, <span class="pl-s">'Encoder and transducer have to have the same amount'</span> \
                                                                <span class="pl-s">'of hidden units'</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">input_dimensions</span> <span class="pl-c1">=</span> <span class="pl-s1">input_dimensions</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span> <span class="pl-c1">=</span> <span class="pl-s1">vocab_ids</span>
        <span class="pl-s1">self</span>.<span class="pl-v">E_SYMBOL</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span>.<span class="pl-en">append</span>(<span class="pl-s">'E_SYMBOL'</span>)
        <span class="pl-s1">self</span>.<span class="pl-v">GO_SYMBOL</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span>.<span class="pl-en">append</span>(<span class="pl-s">'GO_SYMBOL'</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">vocab_size</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">vocab_ids</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">input_embedding_size</span> <span class="pl-c1">=</span> <span class="pl-s1">input_embedding_size</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">inputs_embedded</span> <span class="pl-c1">=</span> <span class="pl-s1">inputs_embedded</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">encoder_hidden_units</span> <span class="pl-c1">=</span> <span class="pl-s1">encoder_hidden_units</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">transducer_hidden_units</span> <span class="pl-c1">=</span> <span class="pl-s1">transducer_hidden_units</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">input_block_size</span> <span class="pl-c1">=</span> <span class="pl-s1">input_block_size</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">beam_width</span> <span class="pl-c1">=</span> <span class="pl-s1">beam_width</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">batch_size</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>  <span class="pl-c"># Cannot be increased, see paper</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">log_prob_init_value</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>

<span class="pl-c"># ----------------- Model ---------------------------------------</span>


<span class="pl-k">class</span> <span class="pl-v">Model</span>(<span class="pl-s1">object</span>):
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">cons_manager</span>):
        <span class="pl-s1">self</span>.<span class="pl-s1">var_list</span> <span class="pl-c1">=</span> []
        <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span> <span class="pl-c1">=</span> <span class="pl-s1">cons_manager</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">max_blocks</span>, <span class="pl-s1">self</span>.<span class="pl-s1">inputs_full_raw</span>, <span class="pl-s1">self</span>.<span class="pl-s1">transducer_list_outputs</span>, <span class="pl-s1">self</span>.<span class="pl-s1">start_block</span>, <span class="pl-s1">self</span>.<span class="pl-s1">encoder_hidden_init</span>,\
            <span class="pl-s1">self</span>.<span class="pl-s1">trans_hidden_init</span>, <span class="pl-s1">self</span>.<span class="pl-s1">logits</span>, <span class="pl-s1">self</span>.<span class="pl-s1">encoder_hidden_state_new</span>, \
            <span class="pl-s1">self</span>.<span class="pl-s1">transducer_hidden_state_new</span>, <span class="pl-s1">self</span>.<span class="pl-s1">train_saver</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">build_full_transducer</span>()

        <span class="pl-s1">self</span>.<span class="pl-s1">targets</span>, <span class="pl-s1">self</span>.<span class="pl-s1">train_op</span>, <span class="pl-s1">self</span>.<span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">build_training_step</span>()

    <span class="pl-k">def</span> <span class="pl-en">build_full_transducer</span>(<span class="pl-s1">self</span>):
        <span class="pl-k">with</span> <span class="pl-s1">tf</span>.<span class="pl-en">variable_scope</span>(<span class="pl-s">'transducer_training'</span>):

            <span class="pl-s1">embeddings</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Variable</span>(<span class="pl-s1">tf</span>.<span class="pl-en">random_uniform</span>([<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">vocab_size</span>,
                                                        <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">input_embedding_size</span>], <span class="pl-c1">-</span><span class="pl-c1">1.0</span>, <span class="pl-c1">1.0</span>),
                                     <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>,
                                     <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'embedding'</span>)
            <span class="pl-c"># Inputs</span>
            <span class="pl-s1">max_blocks</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'max_blocks'</span>)  <span class="pl-c"># total amount of blocks to go through</span>
            <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">inputs_embedded</span> <span class="pl-c1">is</span> <span class="pl-c1">True</span>:
                <span class="pl-s1">input_type</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>
            <span class="pl-k">else</span>:
                <span class="pl-s1">input_type</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>
            <span class="pl-s1">inputs_full_raw</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">None</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">batch_size</span>,
                                                    <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">input_dimensions</span>), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">input_type</span>,
                                             <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'inputs_full_raw'</span>)  <span class="pl-c"># shape [max_time, 1, input_dims]</span>
            <span class="pl-s1">transducer_list_outputs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">None</span>,), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>,
                                                     <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'transducer_list_outputs'</span>)  <span class="pl-c"># amount to output per block</span>
            <span class="pl-s1">start_block</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'transducer_start_block'</span>)  <span class="pl-c"># where to start the input</span>

            <span class="pl-s1">encoder_hidden_init</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>,
                                                 <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'encoder_hidden_init'</span>)
            <span class="pl-s1">trans_hidden_init</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>,
                                               <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'trans_hidden_init'</span>)

            <span class="pl-c"># Temporary constants, maybe changed during inference</span>
            <span class="pl-s1">end_symbol</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">get_variable</span>(<span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'end_symbol'</span>,
                                         <span class="pl-s1">initializer</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">constant_initializer</span>(<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">vocab_size</span>),
                                         <span class="pl-s1">shape</span><span class="pl-c1">=</span>(), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>)

            <span class="pl-c"># Turn inputs into tensor which is easily readable#</span>

            <span class="pl-s1">inputs_full</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">inputs_full_raw</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">input_block_size</span>,
                                                             <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">batch_size</span>,
                                                             <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">input_dimensions</span>])

            <span class="pl-c"># Outputs</span>
            <span class="pl-s1">outputs_ta</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">TensorArray</span>(<span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-s1">max_blocks</span>)

            <span class="pl-s1">init_state</span> <span class="pl-c1">=</span> (<span class="pl-s1">start_block</span>, <span class="pl-s1">outputs_ta</span>, <span class="pl-s1">encoder_hidden_init</span>, <span class="pl-s1">trans_hidden_init</span>)

            <span class="pl-c"># Initiate cells, NOTE: if there is a future error, put these back inside the body function</span>
            <span class="pl-s1">encoder_cell</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">rnn</span>.<span class="pl-v">LSTMCell</span>(<span class="pl-s1">num_units</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>)
            <span class="pl-s1">transducer_cell</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">rnn</span>.<span class="pl-v">LSTMCell</span>(<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>)

            <span class="pl-k">def</span> <span class="pl-en">cond</span>(<span class="pl-s1">current_block</span>, <span class="pl-s1">outputs_int</span>, <span class="pl-s1">encoder_hidden</span>, <span class="pl-s1">trans_hidden</span>):
                <span class="pl-k">return</span> <span class="pl-s1">current_block</span> <span class="pl-c1">&lt;</span> <span class="pl-s1">start_block</span> <span class="pl-c1">+</span> <span class="pl-s1">max_blocks</span>

            <span class="pl-k">def</span> <span class="pl-en">body</span>(<span class="pl-s1">current_block</span>, <span class="pl-s1">outputs_int</span>, <span class="pl-s1">encoder_hidden</span>, <span class="pl-s1">trans_hidden</span>):

                <span class="pl-c"># --------------------- ENCODER ----------------------------------------------------------------------</span>
                <span class="pl-s1">encoder_inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">inputs_full</span>[<span class="pl-s1">current_block</span>]
                <span class="pl-s1">encoder_inputs_length</span> <span class="pl-c1">=</span> [<span class="pl-s1">tf</span>.<span class="pl-en">shape</span>(<span class="pl-s1">encoder_inputs</span>)[<span class="pl-c1">0</span>]]
                <span class="pl-s1">encoder_hidden_state</span> <span class="pl-c1">=</span> <span class="pl-s1">encoder_hidden</span>

                <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">inputs_embedded</span> <span class="pl-c1">is</span> <span class="pl-c1">True</span>:
                    <span class="pl-s1">encoder_inputs_embedded</span> <span class="pl-c1">=</span> <span class="pl-s1">encoder_inputs</span>
                <span class="pl-k">else</span>:
                    <span class="pl-s1">encoder_inputs</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">encoder_inputs</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">batch_size</span>])
                    <span class="pl-s1">encoder_inputs_embedded</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">embedding_lookup</span>(<span class="pl-s1">embeddings</span>, <span class="pl-s1">encoder_inputs</span>)

                <span class="pl-c"># Build model</span>

                <span class="pl-c"># Build previous state</span>
                <span class="pl-s1">encoder_hidden_c</span>, <span class="pl-s1">encoder_hidden_h</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">split</span>(<span class="pl-s1">encoder_hidden_state</span>, <span class="pl-s1">num_or_size_splits</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
                <span class="pl-s1">encoder_hidden_c</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">encoder_hidden_c</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>])
                <span class="pl-s1">encoder_hidden_h</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">encoder_hidden_h</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>])
                <span class="pl-s1">encoder_hidden_state_t</span> <span class="pl-c1">=</span> <span class="pl-v">LSTMStateTuple</span>(<span class="pl-s1">encoder_hidden_c</span>, <span class="pl-s1">encoder_hidden_h</span>)

                <span class="pl-c">#   encoder_outputs: [max_time, batch_size, num_units]</span>
                <span class="pl-s1">encoder_outputs</span>, <span class="pl-s1">encoder_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">dynamic_rnn</span>(
                    <span class="pl-s1">encoder_cell</span>, <span class="pl-s1">encoder_inputs_embedded</span>,
                    <span class="pl-s1">sequence_length</span><span class="pl-c1">=</span><span class="pl-s1">encoder_inputs_length</span>, <span class="pl-s1">time_major</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
                    <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>, <span class="pl-s1">initial_state</span><span class="pl-c1">=</span><span class="pl-s1">encoder_hidden_state_t</span>)

                <span class="pl-c"># Modify output of encoder_hidden_state_new so that it can be fed back in again without problems.</span>
                <span class="pl-s1">encoder_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">concat</span>([<span class="pl-s1">encoder_hidden_state_new</span>.<span class="pl-s1">c</span>, <span class="pl-s1">encoder_hidden_state_new</span>.<span class="pl-s1">h</span>], <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
                <span class="pl-s1">encoder_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">encoder_hidden_state_new</span>,
                                                      <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">2</span>, <span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>])

                <span class="pl-c"># --------------------- TRANSDUCER --------------------------------------------------------------------</span>
                <span class="pl-s1">encoder_raw_outputs</span> <span class="pl-c1">=</span> <span class="pl-s1">encoder_outputs</span>
                <span class="pl-c"># Save/load the state as one tensor, use encoder state as init if this is the first block</span>
                <span class="pl-s1">trans_hidden_state</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">cond</span>(<span class="pl-s1">current_block</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>, <span class="pl-k">lambda</span>: <span class="pl-s1">trans_hidden</span>, <span class="pl-k">lambda</span>: <span class="pl-s1">encoder_hidden_state_new</span>)
                <span class="pl-s1">transducer_amount_outputs</span> <span class="pl-c1">=</span> <span class="pl-s1">transducer_list_outputs</span>[<span class="pl-s1">current_block</span> <span class="pl-c1">-</span> <span class="pl-s1">start_block</span>]

                <span class="pl-c"># Model building</span>
                <span class="pl-s1">helper</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">seq2seq</span>.<span class="pl-v">GreedyEmbeddingHelper</span>(
                    <span class="pl-s1">embedding</span><span class="pl-c1">=</span><span class="pl-s1">embeddings</span>,
                    <span class="pl-s1">start_tokens</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-en">tile</span>([<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-v">GO_SYMBOL</span>],
                                         [<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">batch_size</span>]),  <span class="pl-c"># TODO: check if this looks good</span>
                    <span class="pl-s1">end_token</span><span class="pl-c1">=</span><span class="pl-s1">end_symbol</span>)  <span class="pl-c"># vocab size, so that it doesn't prematurely end the decoding</span>

                <span class="pl-s1">attention_states</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">transpose</span>(<span class="pl-s1">encoder_raw_outputs</span>,
                                                [<span class="pl-c1">1</span>, <span class="pl-c1">0</span>, <span class="pl-c1">2</span>])  <span class="pl-c"># attention_states: [batch_size, max_time, num_units]</span>

                <span class="pl-s1">attention_mechanism</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">seq2seq</span>.<span class="pl-v">LuongAttention</span>(
                    <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">encoder_hidden_units</span>, <span class="pl-s1">attention_states</span>)

                <span class="pl-s1">decoder_cell</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">seq2seq</span>.<span class="pl-v">AttentionWrapper</span>(
                    <span class="pl-s1">transducer_cell</span>,
                    <span class="pl-s1">attention_mechanism</span>,
                    <span class="pl-s1">attention_layer_size</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>)

                <span class="pl-s1">projection_layer</span> <span class="pl-c1">=</span> <span class="pl-s1">layers_core</span>.<span class="pl-v">Dense</span>(<span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">vocab_size</span>, <span class="pl-s1">use_bias</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

                <span class="pl-c"># Build previous state</span>
                <span class="pl-s1">trans_hidden_c</span>, <span class="pl-s1">trans_hidden_h</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">split</span>(<span class="pl-s1">trans_hidden_state</span>, <span class="pl-s1">num_or_size_splits</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
                <span class="pl-s1">trans_hidden_c</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">trans_hidden_c</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>])
                <span class="pl-s1">trans_hidden_h</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">trans_hidden_h</span>, <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>])
                <span class="pl-s1">trans_hidden_state_t</span> <span class="pl-c1">=</span> <span class="pl-v">LSTMStateTuple</span>(<span class="pl-s1">trans_hidden_c</span>, <span class="pl-s1">trans_hidden_h</span>)

                <span class="pl-s1">decoder</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">seq2seq</span>.<span class="pl-v">BasicDecoder</span>(
                    <span class="pl-s1">decoder_cell</span>, <span class="pl-s1">helper</span>,
                    <span class="pl-s1">decoder_cell</span>.<span class="pl-en">zero_state</span>(<span class="pl-c1">1</span>, <span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>).<span class="pl-en">clone</span>(<span class="pl-s1">cell_state</span><span class="pl-c1">=</span><span class="pl-s1">trans_hidden_state_t</span>),
                    <span class="pl-s1">output_layer</span><span class="pl-c1">=</span><span class="pl-s1">projection_layer</span>)

                <span class="pl-s1">outputs</span>, <span class="pl-s1">transducer_hidden_state_new</span>, <span class="pl-s1">_</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">seq2seq</span>.<span class="pl-en">dynamic_decode</span>(<span class="pl-s1">decoder</span>,
                                                                                            <span class="pl-s1">output_time_major</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
                                                                                            <span class="pl-s1">maximum_iterations</span><span class="pl-c1">=</span><span class="pl-s1">transducer_amount_outputs</span>)
                <span class="pl-s1">logits</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs</span>.<span class="pl-s1">rnn_output</span>  <span class="pl-c"># logits of shape [max_time,batch_size,vocab_size]</span>
                <span class="pl-s1">decoder_prediction</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs</span>.<span class="pl-s1">sample_id</span>  <span class="pl-c"># For debugging</span>

                <span class="pl-c"># Modify output of transducer_hidden_state_new so that it can be fed back in again without problems.</span>
                <span class="pl-s1">transducer_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">concat</span>(
                    [<span class="pl-s1">transducer_hidden_state_new</span>[<span class="pl-c1">0</span>].<span class="pl-s1">c</span>, <span class="pl-s1">transducer_hidden_state_new</span>[<span class="pl-c1">0</span>].<span class="pl-s1">h</span>],
                    <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
                <span class="pl-s1">transducer_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(<span class="pl-s1">transducer_hidden_state_new</span>,
                                                         <span class="pl-s1">shape</span><span class="pl-c1">=</span>[<span class="pl-c1">2</span>, <span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">transducer_hidden_units</span>])


                <span class="pl-c"># Note the outputs</span>
                <span class="pl-s1">outputs_int</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs_int</span>.<span class="pl-en">write</span>(<span class="pl-s1">current_block</span> <span class="pl-c1">-</span> <span class="pl-s1">start_block</span>, <span class="pl-s1">logits</span>)

                <span class="pl-k">return</span> <span class="pl-s1">current_block</span> <span class="pl-c1">+</span> <span class="pl-c1">1</span>, <span class="pl-s1">outputs_int</span>, <span class="pl-s1">encoder_hidden_state_new</span>, <span class="pl-s1">transducer_hidden_state_new</span>

            <span class="pl-s1">_</span>, <span class="pl-s1">outputs_final</span>, <span class="pl-s1">encoder_hidden_state_new</span>, <span class="pl-s1">transducer_hidden_state_new</span> <span class="pl-c1">=</span> \
                <span class="pl-s1">tf</span>.<span class="pl-en">while_loop</span>(<span class="pl-s1">cond</span>, <span class="pl-s1">body</span>, <span class="pl-s1">init_state</span>, <span class="pl-s1">parallel_iterations</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)

            <span class="pl-c"># Process outputs</span>
            <span class="pl-s1">outputs</span> <span class="pl-c1">=</span> <span class="pl-s1">outputs_final</span>.<span class="pl-en">concat</span>()
            <span class="pl-s1">logits</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reshape</span>(
                <span class="pl-s1">outputs</span>,
                <span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">vocab_size</span>))  <span class="pl-c"># And now its [max_output_time, batch_size, vocab]</span>

            <span class="pl-c"># For loading the model later on</span>
            <span class="pl-s1">logits</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">identity</span>(<span class="pl-s1">logits</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'logits'</span>)
            <span class="pl-s1">encoder_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">identity</span>(<span class="pl-s1">encoder_hidden_state_new</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'encoder_hidden_state_new'</span>)
            <span class="pl-s1">transducer_hidden_state_new</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">identity</span>(<span class="pl-s1">transducer_hidden_state_new</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'transducer_hidden_state_new'</span>)

        <span class="pl-s1">train_saver</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">Saver</span>()  <span class="pl-c"># For now save everything</span>

        <span class="pl-k">return</span> <span class="pl-s1">max_blocks</span>, <span class="pl-s1">inputs_full_raw</span>, <span class="pl-s1">transducer_list_outputs</span>, <span class="pl-s1">start_block</span>, <span class="pl-s1">encoder_hidden_init</span>,\
            <span class="pl-s1">trans_hidden_init</span>, <span class="pl-s1">logits</span>, <span class="pl-s1">encoder_hidden_state_new</span>, <span class="pl-s1">transducer_hidden_state_new</span>, <span class="pl-s1">train_saver</span>

    <span class="pl-k">def</span> <span class="pl-en">build_training_step</span>(<span class="pl-s1">self</span>):
        <span class="pl-s1">targets</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">placeholder</span>(<span class="pl-s1">shape</span><span class="pl-c1">=</span>(<span class="pl-c1">None</span>,), <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">int32</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'targets'</span>)
        <span class="pl-s1">targets_one_hot</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">one_hot</span>(<span class="pl-s1">targets</span>, <span class="pl-s1">depth</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">cons_manager</span>.<span class="pl-s1">vocab_size</span>, <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">tf</span>.<span class="pl-s1">float32</span>)

        <span class="pl-s1">targets_one_hot</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Print</span>(<span class="pl-s1">targets_one_hot</span>, [<span class="pl-s1">targets</span>], <span class="pl-s1">message</span><span class="pl-c1">=</span><span class="pl-s">'Targets: '</span>, <span class="pl-s1">summarize</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)
        <span class="pl-s1">targets_one_hot</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-v">Print</span>(<span class="pl-s1">targets_one_hot</span>, [<span class="pl-s1">tf</span>.<span class="pl-en">argmax</span>(<span class="pl-s1">self</span>.<span class="pl-s1">logits</span>, <span class="pl-s1">axis</span><span class="pl-c1">=</span><span class="pl-c1">2</span>)], <span class="pl-s1">message</span><span class="pl-c1">=</span><span class="pl-s">'Argmax: '</span>, <span class="pl-s1">summarize</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)

        <span class="pl-s1">stepwise_cross_entropy</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">nn</span>.<span class="pl-en">softmax_cross_entropy_with_logits</span>(<span class="pl-s1">labels</span><span class="pl-c1">=</span><span class="pl-s1">targets_one_hot</span>,
                                                                         <span class="pl-s1">logits</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">logits</span>)
        <span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-en">reduce_mean</span>(<span class="pl-s1">stepwise_cross_entropy</span>)
        <span class="pl-s1">train_op</span> <span class="pl-c1">=</span> <span class="pl-s1">tf</span>.<span class="pl-s1">train</span>.<span class="pl-v">AdamOptimizer</span>().<span class="pl-en">minimize</span>(<span class="pl-s1">loss</span>)
        <span class="pl-k">return</span> <span class="pl-s1">targets</span>, <span class="pl-s1">train_op</span>, <span class="pl-s1">loss</span>


<span class="pl-s1">constants_manager</span> <span class="pl-c1">=</span> <span class="pl-v">ConstantsManager</span>(<span class="pl-s1">input_dimensions</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">input_embedding_size</span><span class="pl-c1">=</span><span class="pl-c1">11</span>, <span class="pl-s1">inputs_embedded</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
                                     <span class="pl-s1">encoder_hidden_units</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">transducer_hidden_units</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">vocab_ids</span><span class="pl-c1">=</span>[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>],
                                     <span class="pl-s1">input_block_size</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">beam_width</span><span class="pl-c1">=</span><span class="pl-c1">5</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">Model</span>(<span class="pl-s1">cons_manager</span><span class="pl-c1">=</span><span class="pl-s1">constants_manager</span>)</pre></div>
<p dir="auto">I can try and make a smaller fail case if needed.</p>
<p dir="auto">Thanks!<br>
Nikita</p>