<p dir="auto">Hi,</p>
<p dir="auto">For example, I'd like to insert some new layers to VGG model before the dense layers, load the parameters, freeze them and continue training.</p>
<p dir="auto">I followed some old issues, which are popping up the top dense and outupt layers, adding new layers and the dense and output layers again.</p>
<p dir="auto">When I try to add new layers, I failed with this error:<br>
Exception: Input 0 is incompatible with layer pool6_zp: expected ndim=4, found ndim=2</p>
<p dir="auto">Actually I also tried just add back the flatten layer and failed too.<br>
I'm using Keras 1.0.4 with tensorflow as backend.</p>
<p dir="auto">I'm really confused. How can I do that?</p>
<p dir="auto">Thanks a lot!</p>
<p dir="auto">Here is the test code:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from keras.layers import (AveragePooling2D, Convolution2D, Dense, Dropout,
    Flatten, Input, MaxPooling2D, ZeroPadding2D)
from keras.models import Model, Sequential

vgg_model = Sequential()
vgg_model.add(Convolution2D(64, 3, 3, activation='relu', input_shape=(3, 224, 224), border_mode='same', name='conv1_1'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv1_1_zp'))
vgg_model.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool1'))

vgg_model.add(ZeroPadding2D((1, 1), name='pool1_zp'))
vgg_model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv2_1_zp'))
vgg_model.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool2'))

vgg_model.add(ZeroPadding2D((1, 1), name='pool2_zp'))
vgg_model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv3_1_zp'))
vgg_model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv3_2_zp'))
vgg_model.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool3'))

vgg_model.add(ZeroPadding2D((1, 1), name='conv4_1_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv4_2_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool4'))

vgg_model.add(ZeroPadding2D((1, 1), name='pool4_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv5_1_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv5_2_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool5'))

vgg_model.add(Flatten())
vgg_model.add(Dense(4096, activation='relu', name='fc6'))
vgg_model.add(Dropout(0.5, name='fc6do'))
vgg_model.add(Dense(4096, activation='relu', name='fc7'))
vgg_model.add(Dropout(0.5, name='fc7do'))
vgg_model.add(Dense(1000, activation='softmax', name='vgg_classes'))

del vgg_model.layers[-6:]
'''
vgg_model.add(ZeroPadding2D((1, 1), name='pool6_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_1'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv6_1_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_2'))
vgg_model.add(ZeroPadding2D((1, 1), name='conv6_2_zp'))
vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_3'))
vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool5'))
'''
vgg_model.add(Flatten())
vgg_model.add(Dense(4096, activation='relu', name='fc6'))
vgg_model.add(Dropout(0.5, name='fc6do'))
vgg_model.add(Dense(4096, activation='relu', name='fc7'))
vgg_model.add(Dropout(0.5, name='fc7do'))
vgg_model.add(Dense(1000, activation='softmax', name='vgg_classes'))

"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">keras</span>.<span class="pl-s1">layers</span> <span class="pl-k">import</span> (<span class="pl-v">AveragePooling2D</span>, <span class="pl-v">Convolution2D</span>, <span class="pl-v">Dense</span>, <span class="pl-v">Dropout</span>,
    <span class="pl-v">Flatten</span>, <span class="pl-v">Input</span>, <span class="pl-v">MaxPooling2D</span>, <span class="pl-v">ZeroPadding2D</span>)
<span class="pl-k">from</span> <span class="pl-s1">keras</span>.<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">Model</span>, <span class="pl-v">Sequential</span>

<span class="pl-s1">vgg_model</span> <span class="pl-c1">=</span> <span class="pl-v">Sequential</span>()
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">64</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">input_shape</span><span class="pl-c1">=</span>(<span class="pl-c1">3</span>, <span class="pl-c1">224</span>, <span class="pl-c1">224</span>), <span class="pl-s1">border_mode</span><span class="pl-c1">=</span><span class="pl-s">'same'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv1_1'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv1_1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">64</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv1_2'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">MaxPooling2D</span>((<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">strides</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool1'</span>))

<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">128</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv2_1'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv2_1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">128</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv2_2'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">MaxPooling2D</span>((<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">strides</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool2'</span>))

<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool2_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">256</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv3_1'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv3_1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">256</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv3_2'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv3_2_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">256</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv3_3'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">MaxPooling2D</span>((<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">strides</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool3'</span>))

<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv4_1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">512</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv4_2'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv4_2_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">512</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv4_3'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">MaxPooling2D</span>((<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">strides</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool4'</span>))

<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool4_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">512</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv5_1'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv5_1_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">512</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv5_2'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">ZeroPadding2D</span>((<span class="pl-c1">1</span>, <span class="pl-c1">1</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv5_2_zp'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Convolution2D</span>(<span class="pl-c1">512</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'conv5_3'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">MaxPooling2D</span>((<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">strides</span><span class="pl-c1">=</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'pool5'</span>))

<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Flatten</span>())
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">4096</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc6'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dropout</span>(<span class="pl-c1">0.5</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc6do'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">4096</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc7'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dropout</span>(<span class="pl-c1">0.5</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc7do'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">1000</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'softmax'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'vgg_classes'</span>))

<span class="pl-k">del</span> <span class="pl-s1">vgg_model</span>.<span class="pl-s1">layers</span>[<span class="pl-c1">-</span><span class="pl-c1">6</span>:]
<span class="pl-s">'''</span>
<span class="pl-s">vgg_model.add(ZeroPadding2D((1, 1), name='pool6_zp'))</span>
<span class="pl-s">vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_1'))</span>
<span class="pl-s">vgg_model.add(ZeroPadding2D((1, 1), name='conv6_1_zp'))</span>
<span class="pl-s">vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_2'))</span>
<span class="pl-s">vgg_model.add(ZeroPadding2D((1, 1), name='conv6_2_zp'))</span>
<span class="pl-s">vgg_model.add(Convolution2D(512, 3, 3, activation='relu', name='conv6_3'))</span>
<span class="pl-s">vgg_model.add(MaxPooling2D((2, 2), strides=(2, 2), name='pool5'))</span>
<span class="pl-s">'''</span>
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Flatten</span>())
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">4096</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc6'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dropout</span>(<span class="pl-c1">0.5</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc6do'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">4096</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'relu'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc7'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dropout</span>(<span class="pl-c1">0.5</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'fc7do'</span>))
<span class="pl-s1">vgg_model</span>.<span class="pl-en">add</span>(<span class="pl-v">Dense</span>(<span class="pl-c1">1000</span>, <span class="pl-s1">activation</span><span class="pl-c1">=</span><span class="pl-s">'softmax'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'vgg_classes'</span>))
</pre></div>
<p dir="auto">i want to apply timedistributed to dropout - the reason is i want each timestep to have exactly x% of the features dropped.</p>
<p dir="auto">i am also using batch_shape for my input with a given batch size since this is required for stateful rnn in my full code. when batch size is given the timedistributed wrapper uses rnn. my backend is theano</p>
<p dir="auto">this gist recreates the issue<br>
<a href="https://gist.github.com/eyaler/6801eee9b2134bcd518ffd34004c1475">https://gist.github.com/eyaler/6801eee9b2134bcd518ffd34004c1475</a></p>
<p dir="auto">may be related to:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="178342859" data-permission-text="Title is private" data-url="https://github.com/keras-team/keras/issues/3834" data-hovercard-type="issue" data-hovercard-url="/keras-team/keras/issues/3834/hovercard" href="https://github.com/keras-team/keras/issues/3834">#3834</a></p>
<p dir="auto">see first comment for error message</p>