@ptrblck
Well I’m also not quite familiar with tensorflow myself,  and that’s the reason why I want to convert this model to pytorch in the first place.
In fact after looking into tensorflow’s doc, I think the running mean and variance
are not updated in the github’s repo, and they are not in the ‘trainable variables’ by default in tensorflow. These lines of code are missing.
  update_ops = tf.compat.v1.get_collection(tf.GraphKeys.UPDATE_OPS)
  with tf.control_dependencies(update_ops):
    train_op = optimizer.minimize(loss)

What’s more , I load the pretrained model provided by the author in tensorflow, and it gives me a reasonable performance. And in fact, I can print these weights of the first batch norm layer with the following lines of code:
bn_weight = tf.get_collection(key=tf.GraphKeys.GLOBAL_VARIABLES, scope="ResidualBlock_1D_0/BatchNorm/    ")
sess.run(bn_weight)

And here’s the output
(Pdb) print(bn_weight)
[<tf.Variable 'ResidualBlock_1D_0/BatchNorm/beta:0' shape=(28,) dtype=float32_ref>, <tf.Variable 'ResidualBlock_1D_0/BatchNorm/moving_mean:0' shape=(28,) dtype=float32_ref>, <tf.Variable 'ResidualBlock_1D_0/BatchNorm/moving_variance:0' shape=(28,) dtype=float32_ref>]
(Pdb) sess.run(bn_weight)
[array([ 0.00602053,  0.0050063 ,  0.00634659,  0.00489874, -0.00110139,
       -0.0022672 ,  0.00327282,  0.00230329, -0.00308617,  0.0024446 ,
        0.00504038,  0.00412193, -0.00564392,  0.00558108,  0.00340598,
        0.00164128, -0.00063161,  0.00081446, -0.00207535,  0.00168148,
       -0.00086416,  0.00218814,  0.00166753, -0.00242493,  0.00044503,
       -0.00097768,  0.00121772,  0.00186761], dtype=float32), array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32), array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)]

And I think I’m doing the right thing in pytorch by setting the mean to zeros and variance to ones. However, the output is just different, wired.
I also tried to set track_running_stats to False or affine to True in pytorch, no help at all.