I noticed that batchNorm2d with affine parameter true is using learnable parameters for each channel (64) instead of each input activation (64x32x32), I guess thats intentional to reduce number of parameters and I am missing something ?
Number of Input Channels = 64
module.bn1.weight: 0.6789193153381348
module.bn1.weight.size(): torch.Size([64])
module.bn1.weight.Gradients: 2.592428207397461
module.bn1.bias.: 0.7608728408813477
module.bn1.bias.size(): torch.Size([64])
module.bn1.bias.Gradients: 1.4683836698532104
I am using the following resent model from -->

  
      github.com
  
  
    kuangliu/pytorch-cifar/blob/master/models/resnet.py 17
'''ResNet in PyTorch.

For Pre-activation ResNet, see 'preact_resnet.py'.

Reference:
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
    Deep Residual Learning for Image Recognition. arXiv:1512.03385
'''
import torch
import torch.nn as nn
import torch.nn.functional as F

from torch.autograd import Variable


class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()


  This file has been truncated. show original