<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using example directory):</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  CentOS7.3</li>
<li>TensorFlow backend (yes / no):  yes</li>
<li>TensorFlow version:  v1.14.0-rc1-22-gaf24dc91b5 1.14.0</li>
<li>Keras version:  2.3.1</li>
<li>Python version:  3.6</li>
<li>CUDA/cuDNN version:</li>
<li>GPU model and memory:</li>
</ul>
<p dir="auto">You can obtain the TensorFlow version with:<br>
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"<br>
You can obtain the Keras version with:<br>
python -c 'import keras as k; print(k.<strong>version</strong>)'</p>
<p dir="auto"><strong>Describe the current behavior</strong><br>
I want concatenate two model to a fully connected layer，and then train it，but keras.fit reporte an assert error like this:<br>
==&gt; Training model...<br>
Traceback (most recent call last):<br>
File "train-2dcnn-1dcnn.py", line 295, in <br>
main()<br>
File "train-2dcnn-1dcnn.py", line 219, in main<br>
model.fit([x_train_1dcnn, x_train_2dcnn], y_train, batch_size=args.batch_size, epochs=args.epochs, validation_split=0.1, callbacks=callback_lists)<br>
File "/usr/local/anaconda3/envs/tensorflow-1.0/lib/python3.6/site-packages/keras/engine/training.py", line 1154, in fit<br>
batch_size=batch_size)<br>
File "/usr/local/anaconda3/envs/tensorflow-1.0/lib/python3.6/site-packages/keras/engine/training.py", line 504, in _standardize_user_data<br>
self._set_inputs(x)<br>
File "/usr/local/anaconda3/envs/tensorflow-1.0/lib/python3.6/site-packages/keras/engine/training.py", line 414, in _set_inputs<br>
assert len(inputs) == 1<br>
AssertionError</p>
<p dir="auto"><strong>Describe the expected behavior</strong><br>
No error occur</p>
<p dir="auto"><strong>Code to reproduce the issue</strong><br>
Provide a reproducible test case that is the bare minimum necessary to generate the problem.<br>
import keras<br>
from keras import layers<br>
from keras.models import Model<br>
from keras.utils import to_categorical<br>
from keras.models import load_model<br>
from keras.callbacks import Callback, ModelCheckpoint, TensorBoard, LearningRateScheduler, EarlyStopping<br>
import keras.backend as K</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="print('==&gt; Building model...')
model1 = keras.Sequential()
model1.add(layers.Conv1D(32, 3, activation='relu', input_shape=(1, 39), padding='same', name='data1'))
model1.add(layers.MaxPooling1D(2, 2, padding='same'))
model1.add(layers.Conv1D(64, 3, activation='relu', padding='same'))
model1.add(layers.MaxPooling1D(2, 2, padding='same'))
model1.add(layers.Flatten())
model1.build((None, 1, 39))

model2 = keras.Sequential()
model2.add(layers.Conv2D(32, 5, activation='relu', input_shape=(28, 28, 3), padding='same', name=&quot;data2&quot;))
model2.add(layers.MaxPooling2D((2, 2), (2, 2), padding='same'))
model2.add(layers.Conv2D(64, 5, activation='relu', padding='same'))
model2.add(layers.MaxPooling2D(2, 2, padding='same'))
model2.add(layers.Conv2D(64, 7, activation='relu', padding='same'))
model2.add(layers.Flatten())
model2.build((None, 28, 28, 3))

model = keras.Sequential()
model.add(layers.Concatenate([model1.output, model2.output]))
model.add(layers.Dense(1024, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(len(class_names), activation='softmax', name=&quot;output&quot;))
model.compile(optimizer=keras.optimizers.RMSprop(),
         loss='sparse_categorical_crossentropy',
         metrics=['accuracy'])
print('==&gt; Training model...')
filepath = &quot;little_class_2dcnn_1dcnn_tf2.h5&quot;
checkpoint = ModelCheckpoint(
    filepath, monitor='val_accuracy', verbose=0, save_best_only=True, mode='max')
lr_scheduler = LearningRateScheduler(schedule)
callback_lists = [lr_scheduler, checkpoint]
model.fit([x_train_1dcnn, x_train_2dcnn], y_train, batch_size=args.batch_size, epochs=args.epochs, validation_split=0.1, callbacks=callback_lists)"><pre class="notranslate"><code class="notranslate">print('==&gt; Building model...')
model1 = keras.Sequential()
model1.add(layers.Conv1D(32, 3, activation='relu', input_shape=(1, 39), padding='same', name='data1'))
model1.add(layers.MaxPooling1D(2, 2, padding='same'))
model1.add(layers.Conv1D(64, 3, activation='relu', padding='same'))
model1.add(layers.MaxPooling1D(2, 2, padding='same'))
model1.add(layers.Flatten())
model1.build((None, 1, 39))

model2 = keras.Sequential()
model2.add(layers.Conv2D(32, 5, activation='relu', input_shape=(28, 28, 3), padding='same', name="data2"))
model2.add(layers.MaxPooling2D((2, 2), (2, 2), padding='same'))
model2.add(layers.Conv2D(64, 5, activation='relu', padding='same'))
model2.add(layers.MaxPooling2D(2, 2, padding='same'))
model2.add(layers.Conv2D(64, 7, activation='relu', padding='same'))
model2.add(layers.Flatten())
model2.build((None, 28, 28, 3))

model = keras.Sequential()
model.add(layers.Concatenate([model1.output, model2.output]))
model.add(layers.Dense(1024, activation='relu'))
model.add(layers.Dense(128, activation='relu'))
model.add(layers.Dense(len(class_names), activation='softmax', name="output"))
model.compile(optimizer=keras.optimizers.RMSprop(),
         loss='sparse_categorical_crossentropy',
         metrics=['accuracy'])
print('==&gt; Training model...')
filepath = "little_class_2dcnn_1dcnn_tf2.h5"
checkpoint = ModelCheckpoint(
    filepath, monitor='val_accuracy', verbose=0, save_best_only=True, mode='max')
lr_scheduler = LearningRateScheduler(schedule)
callback_lists = [lr_scheduler, checkpoint]
model.fit([x_train_1dcnn, x_train_2dcnn], y_train, batch_size=args.batch_size, epochs=args.epochs, validation_split=0.1, callbacks=callback_lists)
</code></pre></div>
<p dir="auto">The dataset shape is as follows：<br>
x_train_1dcnn.shape:  (322819, 1, 39)<br>
x_train_2dcnn.shape:  (322819, 28, 28, 3)<br>
y_train.shape:  (322819, 1)</p>
<p dir="auto"><strong>Other info / logs</strong><br>
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.</p>
<p dir="auto"><strong>Important note:</strong> the development of Keras is currently taking place at <a href="https://github.com/tensorflow/tensorflow"><code class="notranslate">github.com/tensorflow/tensorflow</code></a>. Please file your bug report by creating a new issue in the TensorFlow repository.</p>
<p dir="auto">Please note that multi-backend Keras development has been discontinued. Do not report issues about multi-backend Keras (Keras 2.3.1 and lower), only report issues about the TensorFlow implementation of Keras (tf.keras).</p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04</li>
<li>Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:</li>
<li>TensorFlow installed from (source or binary): pip</li>
<li>TensorFlow version (use command below): 2.2</li>
<li>Python version: 3.6.10</li>
<li>Bazel version (if compiling from source):</li>
<li>GCC/Compiler version (if compiling from source):</li>
<li>CUDA/cuDNN version: CUDA 10.1, cuDNN 7.6.5</li>
<li>GPU model and memory: GeForce GTX Titan X, 12 GB</li>
</ul>
<p dir="auto"><strong>Describe the current behavior</strong></p>
<p dir="auto">After migrating some Keras code from Theano backend (Keras 2.0.7, Theano 1.0.0) to pure TensorFlow, the model is unable to train with the same success as it did before. I did not change anything about the model architecture, hyper parameters, or parameters in the fit function. All that changed were import statements (ex: keras.layers -&gt; tensorflow.keras.layers)</p>
<p dir="auto">With TensorFlow:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2020-07-10 23:12:37 WARNING  Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the tf.data.Dataset.
Epoch 1/8
2020-07-10 23:12:38.795744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-10 23:12:38 WARNING  multiprocessing can interact badly with TensorFlow, causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended.
2020-07-10 23:12:39.211689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
100/100 [==============================] - 10s 103ms/step - loss: 5.1734 - accuracy: 0.1002
Epoch 2/8
100/100 [==============================] - 10s 102ms/step - loss: 6.1694 - accuracy: 0.1079
Epoch 3/8
100/100 [==============================] - 10s 102ms/step - loss: 13.4283 - accuracy: 0.0830
Epoch 4/8
100/100 [==============================] - 10s 101ms/step - loss: 22.0164 - accuracy: 0.0838
Epoch 5/8
100/100 [==============================] - 10s 102ms/step - loss: 44.7066 - accuracy: 0.0575
Epoch 6/8
100/100 [==============================] - 10s 102ms/step - loss: 38.3269 - accuracy: 0.0690
Epoch 7/8
100/100 [==============================] - 10s 102ms/step - loss: 53.5934 - accuracy: 0.0641
Epoch 8/8
100/100 [==============================] - 10s 101ms/step - loss: 66.5487 - accuracy: 0.0673"><pre class="notranslate"><code class="notranslate">2020-07-10 23:12:37 WARNING  Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the tf.data.Dataset.
Epoch 1/8
2020-07-10 23:12:38.795744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-10 23:12:38 WARNING  multiprocessing can interact badly with TensorFlow, causing nondeterministic deadlocks. For high performance data pipelines tf.data is recommended.
2020-07-10 23:12:39.211689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
100/100 [==============================] - 10s 103ms/step - loss: 5.1734 - accuracy: 0.1002
Epoch 2/8
100/100 [==============================] - 10s 102ms/step - loss: 6.1694 - accuracy: 0.1079
Epoch 3/8
100/100 [==============================] - 10s 102ms/step - loss: 13.4283 - accuracy: 0.0830
Epoch 4/8
100/100 [==============================] - 10s 101ms/step - loss: 22.0164 - accuracy: 0.0838
Epoch 5/8
100/100 [==============================] - 10s 102ms/step - loss: 44.7066 - accuracy: 0.0575
Epoch 6/8
100/100 [==============================] - 10s 102ms/step - loss: 38.3269 - accuracy: 0.0690
Epoch 7/8
100/100 [==============================] - 10s 102ms/step - loss: 53.5934 - accuracy: 0.0641
Epoch 8/8
100/100 [==============================] - 10s 101ms/step - loss: 66.5487 - accuracy: 0.0673
</code></pre></div>
<p dir="auto"><strong>Describe the expected behavior</strong></p>
<p dir="auto">Expected training behavior should be similar to that of Keras with Theano backend:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="/usr/local/lib/python3.6/site-packages/keras/engine/training.py:1984: UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the keras.utils.Sequence class.
  UserWarning('Using a generator with use_multiprocessing=True')
Epoch 1/8
100/100 [==============================] - 14s - loss: 4.8412 - acc: 0.1197          
Epoch 2/8
100/100 [==============================] - 13s - loss: 3.7391 - acc: 0.2454     
Epoch 3/8
100/100 [==============================] - 13s - loss: 3.2523 - acc: 0.3528     
Epoch 4/8
100/100 [==============================] - 13s - loss: 2.9821 - acc: 0.3923     
Epoch 5/8
100/100 [==============================] - 13s - loss: 2.8871 - acc: 0.4059     
Epoch 6/8
100/100 [==============================] - 13s - loss: 2.8087 - acc: 0.4177     
Epoch 7/8
100/100 [==============================] - 13s - loss: 2.7203 - acc: 0.4290     
Epoch 8/8
100/100 [==============================] - 13s - loss: 2.6507 - acc: 0.4351"><pre class="notranslate"><code class="notranslate">/usr/local/lib/python3.6/site-packages/keras/engine/training.py:1984: UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using the keras.utils.Sequence class.
  UserWarning('Using a generator with use_multiprocessing=True')
Epoch 1/8
100/100 [==============================] - 14s - loss: 4.8412 - acc: 0.1197          
Epoch 2/8
100/100 [==============================] - 13s - loss: 3.7391 - acc: 0.2454     
Epoch 3/8
100/100 [==============================] - 13s - loss: 3.2523 - acc: 0.3528     
Epoch 4/8
100/100 [==============================] - 13s - loss: 2.9821 - acc: 0.3923     
Epoch 5/8
100/100 [==============================] - 13s - loss: 2.8871 - acc: 0.4059     
Epoch 6/8
100/100 [==============================] - 13s - loss: 2.8087 - acc: 0.4177     
Epoch 7/8
100/100 [==============================] - 13s - loss: 2.7203 - acc: 0.4290     
Epoch 8/8
100/100 [==============================] - 13s - loss: 2.6507 - acc: 0.4351
</code></pre></div>
<p dir="auto"><strong>Standalone code to reproduce the issue</strong></p>
<p dir="auto">Model architecture:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import tensorflow.keras.backend as K
from tensorflow.keras import Model, Sequential, Input
from tensorflow.keras.layers import Dense, Lambda, Embedding, LSTM
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.metrics import Accuracy

epoch = 1
batch_size = 2048
validation_batch_size = 1024
one_hot_dim = 80000
embedding_dim = 128
max_seq_len = 50
hidden_layer_dim = 512
max_q_size = 5000

nb_worker = 5
one_indexing = False
featurizer = &quot;sequence&quot;
featurizer_parameters = [one_hot_dim, max_seq_len, one_indexing]
def create_model_architecture(num_classes):
    # build model architecture
    model = Sequential(
         [
             Input(shape=(max_seq_len,), name=&quot;query_input&quot;),
             Embedding(output_dim=embedding_dim, input_dim=one_hot_dim, input_length=max_seq_len),
             LSTM(embedding_dim, input_shape=(max_seq_len, embedding_dim)),
             Lambda(lambda x: K.cast(x, &quot;float32&quot;)),
             Dense(units=hidden_layer_dim, kernel_initializer=&quot;he_normal&quot;, activation=&quot;relu&quot;),
             Dense(units=num_classes, kernel_initializer=&quot;normal&quot;, activation=&quot;softmax&quot;, name=&quot;prediction&quot;)
         ]
    )
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']
    model.summary()
    return model"><pre class="notranslate"><code class="notranslate">import tensorflow.keras.backend as K
from tensorflow.keras import Model, Sequential, Input
from tensorflow.keras.layers import Dense, Lambda, Embedding, LSTM
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.metrics import Accuracy

epoch = 1
batch_size = 2048
validation_batch_size = 1024
one_hot_dim = 80000
embedding_dim = 128
max_seq_len = 50
hidden_layer_dim = 512
max_q_size = 5000

nb_worker = 5
one_indexing = False
featurizer = "sequence"
featurizer_parameters = [one_hot_dim, max_seq_len, one_indexing]
def create_model_architecture(num_classes):
    # build model architecture
    model = Sequential(
         [
             Input(shape=(max_seq_len,), name="query_input"),
             Embedding(output_dim=embedding_dim, input_dim=one_hot_dim, input_length=max_seq_len),
             LSTM(embedding_dim, input_shape=(max_seq_len, embedding_dim)),
             Lambda(lambda x: K.cast(x, "float32")),
             Dense(units=hidden_layer_dim, kernel_initializer="he_normal", activation="relu"),
             Dense(units=num_classes, kernel_initializer="normal", activation="softmax", name="prediction")
         ]
    )
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']
    model.summary()
    return model
</code></pre></div>
<p dir="auto">In model.compile, I have tried also using the classes for optimizer, loss, and metrics (such as Adam, CategoricalCrossentropy, Accuracy). It causes the accuracy to be much higher but the loss still increases exponentially.</p>
<p dir="auto">Model fit call:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="model.fit(data_generator_train.generate(),
            steps_per_epoch=args.step_log,
            epochs=8,
            validation_data=data_generator_validate.generate() if data_generator_validate else None,
            validation_steps=data_generator_validate.num_batches if data_generator_validate else None,
            max_queue_size=model_module.max_q_size,
            shuffle=True,
            use_multiprocessing=True,
            workers=model_module.nb_worker,
            callbacks=[check_pointer])"><pre class="notranslate"><code class="notranslate">model.fit(data_generator_train.generate(),
            steps_per_epoch=args.step_log,
            epochs=8,
            validation_data=data_generator_validate.generate() if data_generator_validate else None,
            validation_steps=data_generator_validate.num_batches if data_generator_validate else None,
            max_queue_size=model_module.max_q_size,
            shuffle=True,
            use_multiprocessing=True,
            workers=model_module.nb_worker,
            callbacks=[check_pointer])
</code></pre></div>
<p dir="auto">Data generator function:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="@threadsafe_generator
def generate(self):
   while True:
      for filename in glob.glob(self.dataset_files):
         with open(filename) as fr:
            while True:
               lines = list(islice(fr, self.batch_size))
               if not lines:
                  break
               x, y, w = self.get_train_vectors(lines)
               yield x, y, w"><pre class="notranslate"><code class="notranslate">@threadsafe_generator
def generate(self):
   while True:
      for filename in glob.glob(self.dataset_files):
         with open(filename) as fr:
            while True:
               lines = list(islice(fr, self.batch_size))
               if not lines:
                  break
               x, y, w = self.get_train_vectors(lines)
               yield x, y, w
</code></pre></div>
<p dir="auto"><strong>Other info / logs</strong></p>
<p dir="auto">Other things I have tried:</p>
<p dir="auto">Using tensorflow.keras.utils.Sequence instead of a generator, and including shuffle=True<br>
Not using multiprocessing<br>
Not using Sequential<br>
Using classes instead of names in model.compile()<br>
Changing parameters like batch size, steps_per_epoch, etc.</p>
<p dir="auto">Other notes:</p>
<p dir="auto">I am using a custom Docker image to run the model. I have made volume mountings from the local machine, including CUDA, cuDNN, etc. under /usr/local/cuda. TensorFlow 2.2 also has a library (libcublas.so) which is not located there, so I have an additional mapping under /usr/lib/x86_64-linux-gnu.</p>