<p dir="auto">Hi!</p>
<p dir="auto">I want to make a custom layer which is supposed to fuse the output of a Dense Layer with a Convolution2D Layer.</p>
<p dir="auto">The Idea came from <a href="http://dl.acm.org/citation.cfm?id=2925974" rel="nofollow">this paper</a> and here's the network:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/22426131/20623098/5031fff4-b31b-11e6-8594-a54e83fce131.PNG"><img src="https://cloud.githubusercontent.com/assets/22426131/20623098/5031fff4-b31b-11e6-8594-a54e83fce131.PNG" alt="The Network" style="max-width: 100%;"></a></p>
<p dir="auto">the fusion layer tries to fuse the Convolution2D tensor (<code class="notranslate">256x28x28</code>) with the Dense tensor (<code class="notranslate">256</code>). here's the equation for it:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/22426131/20623156/86186496-b31b-11e6-9cb4-bc5baa024e24.PNG"><img src="https://cloud.githubusercontent.com/assets/22426131/20623156/86186496-b31b-11e6-9cb4-bc5baa024e24.PNG" alt="The Fusion Formula" style="max-width: 100%;"></a></p>
<p dir="auto"><code class="notranslate">y_global =&gt; Dense layer output with shape 256</code><br>
<code class="notranslate">y_mid =&gt; Convolution2D layer output with shape 256x28x28</code></p>
<p dir="auto">Here's the description of the paper about the Fusion process:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://cloud.githubusercontent.com/assets/22426131/20623587/843d158e-b31d-11e6-9fa0-49709a241052.PNG"><img src="https://cloud.githubusercontent.com/assets/22426131/20623587/843d158e-b31d-11e6-9fa0-49709a241052.PNG" alt="capture3" style="max-width: 100%;"></a></p>
<p dir="auto">I ended up making a new custom layer like below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class FusionLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(FusionLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        input_dim = input_shape[1][1]
        initial_weight_value = np.random.random((input_dim, self.output_dim))
        self.W = K.variable(initial_weight_value)
        self.b = K.zeros((input_dim,))
        self.trainable_weights = [self.W, self.b]

    def call(self, inputs, mask=None):
        y_global = inputs[0]
        y_mid = inputs[1]
        # the code below should be modified
        output = K.dot(K.concatenate([y_global, y_mid]), self.W)
        output += self.b
        return self.activation(output)

    def get_output_shape_for(self, input_shape):
        assert input_shape and len(input_shape) == 2
        return (input_shape[0], self.output_dim)"><pre class="notranslate"><code class="notranslate">class FusionLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(FusionLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        input_dim = input_shape[1][1]
        initial_weight_value = np.random.random((input_dim, self.output_dim))
        self.W = K.variable(initial_weight_value)
        self.b = K.zeros((input_dim,))
        self.trainable_weights = [self.W, self.b]

    def call(self, inputs, mask=None):
        y_global = inputs[0]
        y_mid = inputs[1]
        # the code below should be modified
        output = K.dot(K.concatenate([y_global, y_mid]), self.W)
        output += self.b
        return self.activation(output)

    def get_output_shape_for(self, input_shape):
        assert input_shape and len(input_shape) == 2
        return (input_shape[0], self.output_dim)
</code></pre></div>
<p dir="auto">I think I got the <code class="notranslate">__init__</code> and <code class="notranslate">build</code> methods right but I don't know how to concatenate <code class="notranslate">y_global</code> (256 dimesnions) with <code class="notranslate">y-mid</code> (256x28x28 dimensions) in the <code class="notranslate">call</code> layer so that the output would be the same as the equation mentioned above.</p>
<p dir="auto">How can I implement this equation in the <code class="notranslate">call</code> method?</p>
<p dir="auto">Thanks so much...</p>
<p dir="auto">UPDATE: any other way to successfully integrate the data of these 2 layers is also acceptable for me... it doesn't exactly have to be the way mentioned in the paper but it needs to at least return an acceptable output...</p>
<p dir="auto">Easy to reproduce this bug when training GANs but probably occurs in other use cases as well. When <code class="notranslate">BatchNormalization</code> is used in the <code class="notranslate">generator</code> of a GAN, <code class="notranslate">combined.train_on_batch</code> fails. It's really weird but for some reason <code class="notranslate">combined.train_on_batch</code> does not calculate the loss correctly when batch norm is used in the generator. I tested the loss by hand doing something like:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="loss = combined.train_on_batch(x, y)
combined_pred = combined.predict(x)
loss_check = K.eval(custom_loss(y, combined_pred))
&gt;&gt;&gt; loss != loss_check"><pre class="notranslate"><span class="pl-s1">loss</span> <span class="pl-c1">=</span> <span class="pl-s1">combined</span>.<span class="pl-en">train_on_batch</span>(<span class="pl-s1">x</span>, <span class="pl-s1">y</span>)
<span class="pl-s1">combined_pred</span> <span class="pl-c1">=</span> <span class="pl-s1">combined</span>.<span class="pl-en">predict</span>(<span class="pl-s1">x</span>)
<span class="pl-s1">loss_check</span> <span class="pl-c1">=</span> <span class="pl-v">K</span>.<span class="pl-en">eval</span>(<span class="pl-en">custom_loss</span>(<span class="pl-s1">y</span>, <span class="pl-s1">combined_pred</span>))
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">loss</span> <span class="pl-c1">!=</span> <span class="pl-s1">loss_check</span></pre></div>
<p dir="auto">You can reproduce this in <code class="notranslate">wGAN</code> branch here: <a href="https://github.com/wayaai/GAN-Sandbox">https://github.com/wayaai/GAN-Sandbox</a> and uncomment the BatchNorm layer in the generator.</p>