<p dir="auto">A number of us have shown interest in implementing seq2seq models in Keras <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="206947870" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/5358" data-hovercard-type="issue" data-hovercard-url="/keras-team/keras/issues/5358/hovercard" href="https://github.com/keras-team/keras/issues/5358">#5358</a>, so I'm creating a central issue for it. We can discuss design decisions and prevent duplication of work here.</p>
<p dir="auto">There are a number of existing pull requests related to seq2seq models:</p>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="210969976" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/5559" data-hovercard-type="pull_request" data-hovercard-url="/keras-team/keras/pull/5559/hovercard" href="https://github.com/keras-team/keras/pull/5559">#5559</a> (Merged) Specify States of RNN Layers Symbolically.
<ul dir="auto">
<li>We have to specify the initial state of the decoder symbolically. This pull request adds the <code class="notranslate">initial_state</code> argument, which is either a tensor or list of tensors, to <code class="notranslate">Recurrent.call</code>.</li>
</ul>
</li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="213621037" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/5731" data-hovercard-type="pull_request" data-hovercard-url="/keras-team/keras/pull/5731/hovercard" href="https://github.com/keras-team/keras/pull/5731">#5731</a> Add <code class="notranslate">return_state</code> flag to RNNs for seq2seq modeling.
<ul dir="auto">
<li>We have to retrieve the final state of encoder. This pull request adds the <code class="notranslate">return_state</code> flag to <code class="notranslate">Recurrent.__init__</code>. If <code class="notranslate">True</code>, the output of the RNN will be a list of tensors. The first tensor is the output, and the remaining tensors are the final state.</li>
</ul>
</li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="213651275" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/5737" data-hovercard-type="pull_request" data-hovercard-url="/keras-team/keras/pull/5737/hovercard" href="https://github.com/keras-team/keras/pull/5737">#5737</a> Implement RNN decoding; feed output t-1 as input t
<ul dir="auto">
<li>During test time, we have to feed the output of the decoder back into itself as the input for the next time step. This pull request adds the <code class="notranslate">output_length</code> argument to <code class="notranslate">Recurrent.__init__</code>. It requires <code class="notranslate">unroll=True</code>. If <code class="notranslate">output_length</code> is greater than the length of the input, the output of that layer will be fed back into the RNN as the input for the next time step.</li>
<li>Note: During training time, we can use teacher forcing, instead of feeding back the outputs of the decoder.</li>
</ul>
</li>
</ul>
<p dir="auto">There is currently a seq2seq example in <code class="notranslate">keras/examples</code>: <code class="notranslate">addition_rnn.py</code>. However, the implementation could be greatly improved. It currently supports only a single layer encoder-decoder architecture. It also does not perform teacher forcing.</p>
<p dir="auto">Feel free to comment on any ideas, questions or comments you might have.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/farizrahman4u/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/farizrahman4u">@farizrahman4u</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fchollet/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fchollet">@fchollet</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/israelg99/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/israelg99">@israelg99</a>.</p>
<p dir="auto">I used code below to combine 2 mobile-net model. After combine i save model as combined.hdf5.<br>
Keras version: 2.2.0 Using TensorFlow backend.<br>
TensorFlow version: 1.8.0</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model_A = load_model('mobilenet_A.hdf5', custom_objects={'relu6': mobilenet.relu6})
model_B = load_model('mobilenet_B.hdf5', custom_objects={'relu6': mobilenet.relu6})
inputs = Input(shape=(224, 224, 3))

pred_A = model_A(inputs)
pred_B = model_B(inputs)

pred_average = keras.layers.Average()([pred_A, pred_B])

model_combined = Model(inputs=inputs, outputs=pred_average)
model_combined.save('combined_test.hdf5')
model_x = load_model('combined_test.hdf5', custom_objects={'relu6': mobilenet.relu6})"><pre class="notranslate"><code class="notranslate">model_A = load_model('mobilenet_A.hdf5', custom_objects={'relu6': mobilenet.relu6})
model_B = load_model('mobilenet_B.hdf5', custom_objects={'relu6': mobilenet.relu6})
inputs = Input(shape=(224, 224, 3))

pred_A = model_A(inputs)
pred_B = model_B(inputs)

pred_average = keras.layers.Average()([pred_A, pred_B])

model_combined = Model(inputs=inputs, outputs=pred_average)
model_combined.save('combined_test.hdf5')
model_x = load_model('combined_test.hdf5', custom_objects={'relu6': mobilenet.relu6})
</code></pre></div>
<p dir="auto">When i load saved model, i get error below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-23-3df9104863a8&gt; in &lt;module&gt;()
      1 model_x = load_model('combined_test.hdf5', 
----&gt; 2                      custom_objects={'relu6': mobilenet.relu6})

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in load_model(filepath, custom_objects, compile)
    262 
    263         # set weights
--&gt; 264         load_weights_from_hdf5_group(f['model_weights'], model.layers)
    265 
    266         if compile:

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in load_weights_from_hdf5_group(f, layers, reshape)
    914                                                        original_keras_version,
    915                                                        original_backend,
--&gt; 916                                                        reshape=reshape)
    917         if len(weight_values) != len(symbolic_weights):
    918             raise ValueError('Layer #' + str(k) +

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    555         weights = convert_nested_time_distributed(weights)
    556     elif layer.__class__.__name__ in ['Model', 'Sequential']:
--&gt; 557         weights = convert_nested_model(weights)
    558 
    559     if original_keras_version == '1':

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in convert_nested_model(weights)
    543                     weights=weights[:num_weights],
    544                     original_keras_version=original_keras_version,
--&gt; 545                     original_backend=original_backend))
    546                 weights = weights[num_weights:]
    547         return new_weights

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    555         weights = convert_nested_time_distributed(weights)
    556     elif layer.__class__.__name__ in ['Model', 'Sequential']:
--&gt; 557         weights = convert_nested_model(weights)
    558 
    559     if original_keras_version == '1':

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in convert_nested_model(weights)
    531                     weights=weights[:num_weights],
    532                     original_keras_version=original_keras_version,
--&gt; 533                     original_backend=original_backend))
    534                 weights = weights[num_weights:]
    535 

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    673             weights[0] = np.reshape(weights[0], layer_weights_shape)
    674         elif layer_weights_shape != weights[0].shape:
--&gt; 675             weights[0] = np.transpose(weights[0], (3, 2, 0, 1))
    676             if layer.__class__.__name__ == 'ConvLSTM2D':
    677                 weights[1] = np.transpose(weights[1], (3, 2, 0, 1))

~/virtualenvs/AILab/lib/python3.5/site-packages/numpy/core/fromnumeric.py in transpose(a, axes)
    548 
    549     &quot;&quot;&quot;
--&gt; 550     return _wrapfunc(a, 'transpose', axes)
    551 
    552 

~/virtualenvs/AILab/lib/python3.5/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     55 def _wrapfunc(obj, method, *args, **kwds):
     56     try:
---&gt; 57         return getattr(obj, method)(*args, **kwds)
     58 
     59     # An AttributeError occurs if the object does not have

ValueError: axes don't match array"><pre class="notranslate"><code class="notranslate">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-23-3df9104863a8&gt; in &lt;module&gt;()
      1 model_x = load_model('combined_test.hdf5', 
----&gt; 2                      custom_objects={'relu6': mobilenet.relu6})

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in load_model(filepath, custom_objects, compile)
    262 
    263         # set weights
--&gt; 264         load_weights_from_hdf5_group(f['model_weights'], model.layers)
    265 
    266         if compile:

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in load_weights_from_hdf5_group(f, layers, reshape)
    914                                                        original_keras_version,
    915                                                        original_backend,
--&gt; 916                                                        reshape=reshape)
    917         if len(weight_values) != len(symbolic_weights):
    918             raise ValueError('Layer #' + str(k) +

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    555         weights = convert_nested_time_distributed(weights)
    556     elif layer.__class__.__name__ in ['Model', 'Sequential']:
--&gt; 557         weights = convert_nested_model(weights)
    558 
    559     if original_keras_version == '1':

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in convert_nested_model(weights)
    543                     weights=weights[:num_weights],
    544                     original_keras_version=original_keras_version,
--&gt; 545                     original_backend=original_backend))
    546                 weights = weights[num_weights:]
    547         return new_weights

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    555         weights = convert_nested_time_distributed(weights)
    556     elif layer.__class__.__name__ in ['Model', 'Sequential']:
--&gt; 557         weights = convert_nested_model(weights)
    558 
    559     if original_keras_version == '1':

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in convert_nested_model(weights)
    531                     weights=weights[:num_weights],
    532                     original_keras_version=original_keras_version,
--&gt; 533                     original_backend=original_backend))
    534                 weights = weights[num_weights:]
    535 

~/virtualenvs/AILab/lib/python3.5/site-packages/keras/engine/saving.py in preprocess_weights_for_loading(layer, weights, original_keras_version, original_backend, reshape)
    673             weights[0] = np.reshape(weights[0], layer_weights_shape)
    674         elif layer_weights_shape != weights[0].shape:
--&gt; 675             weights[0] = np.transpose(weights[0], (3, 2, 0, 1))
    676             if layer.__class__.__name__ == 'ConvLSTM2D':
    677                 weights[1] = np.transpose(weights[1], (3, 2, 0, 1))

~/virtualenvs/AILab/lib/python3.5/site-packages/numpy/core/fromnumeric.py in transpose(a, axes)
    548 
    549     """
--&gt; 550     return _wrapfunc(a, 'transpose', axes)
    551 
    552 

~/virtualenvs/AILab/lib/python3.5/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     55 def _wrapfunc(obj, method, *args, **kwds):
     56     try:
---&gt; 57         return getattr(obj, method)(*args, **kwds)
     58 
     59     # An AttributeError occurs if the object does not have

ValueError: axes don't match array
</code></pre></div>