<p dir="auto">I use the following script to remove the 2 inputs from a siamese network and the output (a cosine distance layer).</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="#!/usr/bin/env python

from dir.model import load_siamese_network
import sys


def load_siamese_network(filepath, include_distance=True):
    custom_objects = {
        'contrastive_loss': contrastive_loss
    }
    model = load_model(filepath, custom_objects=custom_objects)
    if not include_distance:
        # layer 0 and 1 are inputs
        tuned_model = model.layers[2]
        return tuned_model
    return model


src = sys.argv[1]
dst = sys.argv[2]
model = load_siamese_network(src, include_distance=False)

model.save(dst)"><pre class="notranslate"><span class="pl-c">#!/usr/bin/env python</span>

<span class="pl-k">from</span> <span class="pl-s1">dir</span>.<span class="pl-s1">model</span> <span class="pl-k">import</span> <span class="pl-s1">load_siamese_network</span>
<span class="pl-k">import</span> <span class="pl-s1">sys</span>


<span class="pl-k">def</span> <span class="pl-en">load_siamese_network</span>(<span class="pl-s1">filepath</span>, <span class="pl-s1">include_distance</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    <span class="pl-s1">custom_objects</span> <span class="pl-c1">=</span> {
        <span class="pl-s">'contrastive_loss'</span>: <span class="pl-s1">contrastive_loss</span>
    }
    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-en">load_model</span>(<span class="pl-s1">filepath</span>, <span class="pl-s1">custom_objects</span><span class="pl-c1">=</span><span class="pl-s1">custom_objects</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">include_distance</span>:
        <span class="pl-c"># layer 0 and 1 are inputs</span>
        <span class="pl-s1">tuned_model</span> <span class="pl-c1">=</span> <span class="pl-s1">model</span>.<span class="pl-s1">layers</span>[<span class="pl-c1">2</span>]
        <span class="pl-k">return</span> <span class="pl-s1">tuned_model</span>
    <span class="pl-k">return</span> <span class="pl-s1">model</span>


<span class="pl-s1">src</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">argv</span>[<span class="pl-c1">1</span>]
<span class="pl-s1">dst</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">argv</span>[<span class="pl-c1">2</span>]
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-en">load_siamese_network</span>(<span class="pl-s1">src</span>, <span class="pl-s1">include_distance</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

<span class="pl-s1">model</span>.<span class="pl-en">save</span>(<span class="pl-s1">dst</span>)</pre></div>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/416585/32200140-5bfca3e8-bd8d-11e7-86b8-f5fad6471f89.png"><img width="392" alt="screen shot 2017-10-30 at 16 13 53" src="https://user-images.githubusercontent.com/416585/32200140-5bfca3e8-bd8d-11e7-86b8-f5fad6471f89.png" style="max-width: 100%;"></a></p>
<p dir="auto">Here's the code I use to generate the siamese network from a base network: <a href="https://gist.github.com/aeec5b2a6efb1090375d4211aabbed6f">https://gist.github.com/aeec5b2a6efb1090375d4211aabbed6f</a></p>
<p dir="auto">It appears to be working as expected but I was surprised to see that the saved file is twice smaller than the original one (90MB instead of 180MB). How could removing 2 inputs and a distance layer with no weights so dramatically reduce the file size?</p>
<p dir="auto">Since this is a siamese network, both inputs share the same network + weights so removing an input shouldn't really affect the file size that much. That being said, it's possible that Keras isn't that smart about saving the model to disk and is duplicating the network weights for each input although they really are shared.</p>
<p dir="auto">Is this what is happening? Or did I setup the siamese network wrong?</p>
<p dir="auto"><strong>System information</strong></p>
<ul dir="auto">
<li>Have I written custom code (as opposed to using example directory):</li>
<li>OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  CentOS Linux release 7.2.1511 (Core)</li>
<li>TensorFlow backend (yes / no):  YES</li>
<li>TensorFlow version:  v1.12.3-0-g41e0a4f56c 1.12.3</li>
<li>Keras version:  2.1.6-tf</li>
<li>Python version:  python 3.6.5</li>
<li>CUDA/cuDNN version:  9.0.176/7.6.3.30</li>
<li>GPU model and memory:  Tesla P100-PCIE 16GB x4</li>
</ul>
<p dir="auto">My model is shown below, with many BN layers.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" def DnCNN(depth, filters=64, image_channels=1, use_bnorm=True):
    layer_count = 0
    inpt = Input(shape=(None, None, image_channels), name='input' + str(layer_count))
    # 1st layer, Conv+relu
    layer_count += 1
    x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
               name='conv' + str(layer_count))(inpt)
    layer_count += 1
    x = Activation('relu', name='relu' + str(layer_count))(x)
    # depth-2 layers, Conv+BN+relu
    for i in range(depth - 2):
        layer_count += 1
        x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
                   use_bias=False, name='conv' + str(layer_count))(x)
        if use_bnorm:
            layer_count += 1
            # x = BatchNormalization(axis=3, momentum=0.1,epsilon=0.0001, name = 'bn'+str(layer_count))(x)
            x = BatchNormalization(axis=3, momentum=0.0, epsilon=0.0001, name='bn' + str(layer_count))(x)
        layer_count += 1
        x = Activation('relu', name='relu' + str(layer_count))(x)
        # last layer, Conv
    layer_count += 1
    x = Conv2D(filters=image_channels, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal',
               padding='same', use_bias=False, name='conv' + str(layer_count))(x)
    layer_count += 1
    x = Subtract(name='subtract' + str(layer_count))([inpt, x])  # input - noise
    model = Model(inputs=inpt, outputs=x)

    return model "><pre class="notranslate"><code class="notranslate"> def DnCNN(depth, filters=64, image_channels=1, use_bnorm=True):
    layer_count = 0
    inpt = Input(shape=(None, None, image_channels), name='input' + str(layer_count))
    # 1st layer, Conv+relu
    layer_count += 1
    x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
               name='conv' + str(layer_count))(inpt)
    layer_count += 1
    x = Activation('relu', name='relu' + str(layer_count))(x)
    # depth-2 layers, Conv+BN+relu
    for i in range(depth - 2):
        layer_count += 1
        x = Conv2D(filters=filters, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal', padding='same',
                   use_bias=False, name='conv' + str(layer_count))(x)
        if use_bnorm:
            layer_count += 1
            # x = BatchNormalization(axis=3, momentum=0.1,epsilon=0.0001, name = 'bn'+str(layer_count))(x)
            x = BatchNormalization(axis=3, momentum=0.0, epsilon=0.0001, name='bn' + str(layer_count))(x)
        layer_count += 1
        x = Activation('relu', name='relu' + str(layer_count))(x)
        # last layer, Conv
    layer_count += 1
    x = Conv2D(filters=image_channels, kernel_size=(3, 3), strides=(1, 1), kernel_initializer='Orthogonal',
               padding='same', use_bias=False, name='conv' + str(layer_count))(x)
    layer_count += 1
    x = Subtract(name='subtract' + str(layer_count))([inpt, x])  # input - noise
    model = Model(inputs=inpt, outputs=x)

    return model 
</code></pre></div>
<p dir="auto">My test set is (128 * n, 40, 40, 1), batch_size = 128<br>
It's normal for me to run my code on a single GPU<br>
However, when I use multiple GPUs（multi_gpu_num）, the weight of the BN layer becomes NAN.<br>
Strangely, this seems to be related only to GPU_NUM :<br>
When GPU_NUM = 2, the weights become Nan at the 129th step of training,<br>
When GPU_NUM = 4, the weights become Nan at the 65th step of training,<br>
Whether I change the batch_size to 64, 128, or 512, the above behaviors still exist</p>
<p dir="auto">code:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="if __name__ == '__main__':
    # model selection

    # with tf.device('/cpu:0'):
    #     model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    model.summary()

    # for reproducibility
    np.random.seed(seed=10)

    # load the last model in matconvnet style
    initial_epoch = findLastCheckpoint(save_dir=save_dir)
    if initial_epoch &gt; 0:
        print('resuming by loading epoch %03d' % initial_epoch)
        model = load_model(os.path.join(save_dir, 'model_%03d.hdf5' % initial_epoch), compile=False)

    # compile the model
    gpu_num = 2
    print(&quot;[INFO] training with {} GPUs...&quot;.format(gpu_num))
    # model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    parallel_model = multi_gpu_model(model, gpus=gpu_num)
    # model.compile(optimizer=Adam(0.001), loss=sum_squared_error)
    parallel_model.compile(optimizer=Adam(0.001), loss=sum_squared_error)

    # use call back functions
    args.train_data = r'data/Train400' #train set
    args.batch_size = 512
    args.epoch = 2
    checkpointer = ModelCheckpoint(os.path.join(save_dir, 'model_{epoch:03d}.hdf5'),
                                   verbose=1, save_weights_only=False, period=args.save_every)
    csv_logger = CSVLogger(os.path.join(save_dir, 'log.csv'), append=True, separator=',')
    lr_scheduler = LearningRateScheduler(lr_schedule)
    data = train_datagen(batch_size=args.batch_size, data_dir=args.train_data)
    # history = model.fit_generator(data,
    #                               steps_per_epoch=2000, epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
    #                               callbacks=[checkpointer, csv_logger, lr_scheduler])
    history = parallel_model.fit_generator(data,
                                  steps_per_epoch=2000,epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
                                  callbacks=[csv_logger, lr_scheduler])
    model.save('test.h5')
    parallel_model.save('testpa.h5')
"><pre class="notranslate"><code class="notranslate">if __name__ == '__main__':
    # model selection

    # with tf.device('/cpu:0'):
    #     model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    model.summary()

    # for reproducibility
    np.random.seed(seed=10)

    # load the last model in matconvnet style
    initial_epoch = findLastCheckpoint(save_dir=save_dir)
    if initial_epoch &gt; 0:
        print('resuming by loading epoch %03d' % initial_epoch)
        model = load_model(os.path.join(save_dir, 'model_%03d.hdf5' % initial_epoch), compile=False)

    # compile the model
    gpu_num = 2
    print("[INFO] training with {} GPUs...".format(gpu_num))
    # model = DnCNN(depth=17, filters=64, image_channels=1, use_bnorm=True)
    parallel_model = multi_gpu_model(model, gpus=gpu_num)
    # model.compile(optimizer=Adam(0.001), loss=sum_squared_error)
    parallel_model.compile(optimizer=Adam(0.001), loss=sum_squared_error)

    # use call back functions
    args.train_data = r'data/Train400' #train set
    args.batch_size = 512
    args.epoch = 2
    checkpointer = ModelCheckpoint(os.path.join(save_dir, 'model_{epoch:03d}.hdf5'),
                                   verbose=1, save_weights_only=False, period=args.save_every)
    csv_logger = CSVLogger(os.path.join(save_dir, 'log.csv'), append=True, separator=',')
    lr_scheduler = LearningRateScheduler(lr_schedule)
    data = train_datagen(batch_size=args.batch_size, data_dir=args.train_data)
    # history = model.fit_generator(data,
    #                               steps_per_epoch=2000, epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
    #                               callbacks=[checkpointer, csv_logger, lr_scheduler])
    history = parallel_model.fit_generator(data,
                                  steps_per_epoch=2000,epochs=args.epoch, verbose=1, initial_epoch=initial_epoch,
                                  callbacks=[csv_logger, lr_scheduler])
    model.save('test.h5')
    parallel_model.save('testpa.h5')

</code></pre></div>
<p dir="auto">How can I solve this problem？</p>