Hi, I’m trying to convert a tensorflow pretrained model to pytorch.
The network is quite simple, just with one 1dconv and one 1dbn layer
The output is the same after the 1d conv layer. However, I cannot replicate the performance/output after the batchnorm layer
More specifically, there seems to be a bug in the original tensorflow code.
The author seems to forget to add the runningmean/variance of the batchnorm layer in tensorflow to the ‘trainable variables’, so they are never really updated in the model he provided. And the is_train flag is always set to True even in evaluation.
However, when I load the model in pytorch, then set the runningmean to all zeros and running variance to all ones, the output is different nomatter I set model to train or eval.
Nonetheless, the overall performance in the tensorflow model is great. Even if I remove all the bn layers, I can get the same output in my pytorch code, and when I add batchnorm back and retrain the network in pytorch, the performance is bad, about 20 ~ 30 percent lower,
Please check this original code  in tensorflow for details

  
      github.com
  
  
    doubleQ2018/Deep-learning-for-contact_map_v2/blob/master/libs/nets/network.py 4
#!/usr/bin/env python

import tensorflow as tf
import numpy as np

def weight_variable(shape, regularizer, name="W"):
    if regularizer == None:
        initial = tf.truncated_normal(shape, stddev=0.1)
        return tf.Variable(initial, name)
    else:
        return tf.get_variable(name, shape, 
                initializer=tf.random_normal_initializer(), regularizer=regularizer)


def bias_variable(shape, name="b"):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial, name)

### Incoming shape (batch_size, L(seqLen), feature_num)
### Output[:, i, j, :] = incoming[:. i, :] + incoming[:, j, :] + incoming[:, (i+j)/2, :]


  This file has been truncated. show original

  
  
    
    
  
  


And here’s my code for the problem above

  
      github.com
  
  
    YamamotoSayaka/form/blob/master/cvt_bn.py 13
import torch
import torch.nn as nn
import torch.nn.functional as F

import numpy as np
import tensorflow as tf
import pickle

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'

filter_size = 17
in_channels = 26
out_channels = 28
inp = np.random.rand(1,100,26)


# tensorflow model
def weight_variable(shape, name='W'):
    initial = tf.truncated_normal(shape, stddev=0.1)


  This file has been truncated. show original