<p dir="auto">Assume we are trying to learn a sequence to sequence map. For this we can use Recurrent and TimeDistributedDense layers. Now assume that the sequences have different lengths. We should pad both input and desired sequences with zeros, right? But how will the objective function handle the  padded values? There is no choice to pass a mask to the objective function. Won't this bias the cost function?</p>
<p dir="auto">I am trying to implement the SRGAN model from the paper <a href="https://arxiv.org/abs/1609.04802" rel="nofollow">Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network</a>. Since this uses both a GAN and VGG Perceptual losses, I am using modified ActivityRegularizers to incorporate the various losses. I am using the latest Theano and Keras, and the model is for Theano only (I'm using Windows, no Tensorflow yet)</p>
<p dir="auto">The full model architecture and the implementation of the loss are here : <a href="https://gist.github.com/titu1994/abe4bb62da23e5d8c1b3d23ca477f232">SRGAN regularizers gist</a></p>
<p dir="auto">A simplified view of the entire model is:<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/3048602/19019560/e78bfb44-88a9-11e6-864a-720c5e5b2ca0.jpg"><img src="https://cloud.githubusercontent.com/assets/3048602/19019560/e78bfb44-88a9-11e6-864a-720c5e5b2ca0.jpg" alt="srgan-simple-architecture" style="max-width: 100%;"></a></p>
<p dir="auto">In this model, I am passing a 96 x 96 blurred low resolution image (LR) as input to the SR-ResNet network, and 384 x 384 high resolution images (HR) as input 2 and 3 to the Discriminator and VGG networks, as well as the outputs of the SR-ResNet.</p>
<p dir="auto">The issue is that since original inputs to the SR-ResNet have a batch size of 8, Keras assumes that output batch size should also be 8. However, since we are merging the original high resolution images into both the discriminator and VGG networks, batch size becomes 16.</p>
<p dir="auto">The reason for adding the original HR input images as Input2 and Input3 is that the ContentVGGRegularizer needs to compare the gram matrix of the HR inputs to the gram matrix of the generated outputs (G(LR)) from the generative model. Also, to train the discriminator network, we require the original images (D(G(LR)).</p>
<p dir="auto">The error is fairly simple and understandable :</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py&quot;, line 431, in &lt;module&gt;
    srgan_network.pre_train_network(coco_path, nb_epochs=20)
  File &quot;D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py&quot;, line 310, in pre_train_network
    self._train_network(image_dir, nb_epochs=nb_epochs)
  File &quot;D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py&quot;, line 365, in _train_network
    batch_size=self.batch_size // 2,nb_epoch=1, verbose=0)
  File &quot;D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py&quot;, line 1034, in fit
    batch_size=batch_size)
  File &quot;D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py&quot;, line 973, in _standardize_user_data
    check_array_lengths(x, y, sample_weights)
  File &quot;D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py&quot;, line 177, in check_array_lengths
    str(list(set_y)[0]) + ' target samples.')
Exception: Input arrays should have the same number of samples as target arrays. Found 8 input samples and 16 target samples."><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py", line 431, in &lt;module&gt;
    srgan_network.pre_train_network(coco_path, nb_epochs=20)
  File "D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py", line 310, in pre_train_network
    self._train_network(image_dir, nb_epochs=nb_epochs)
  File "D:/Users/Yue/PycharmProjects/Super Resolution using Generative Adversarial Networks/models.py", line 365, in _train_network
    batch_size=self.batch_size // 2,nb_epoch=1, verbose=0)
  File "D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py", line 1034, in fit
    batch_size=batch_size)
  File "D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py", line 973, in _standardize_user_data
    check_array_lengths(x, y, sample_weights)
  File "D:\Users\Yue\Anaconda3\lib\site-packages\keras-1.1.0-py3.4.egg\keras\engine\training.py", line 177, in check_array_lengths
    str(list(set_y)[0]) + ' target samples.')
Exception: Input arrays should have the same number of samples as target arrays. Found 8 input samples and 16 target samples.
</code></pre></div>
<p dir="auto">My question is, is there any way to train this network without manually creating the train_function, getting updates, add regularizers and then use the train_function?</p>
<p dir="auto">Or is there some other way to train such networks without appending the original inputs? I have not yet found a way to mask a portion of the input batch, which could potentially solve this problem (by passing the blurred and HR images to the SR-ResNet input), however this poses another challenge - LR images are 96 x 96, HR images are 384, 384).</p>
<p dir="auto">Any solutions? I can only think of one right now, which is to duplicate most of the _make_train_function and _fit_loop code to bypass this check.</p>