I also added hooks to see if the output till ac3  is same or not for both the models and it turned out to be same.
but when we pass through that particular layer then the change happens.
I also tried to see if it happens when I create a layer seperately
md = nn.Conv2d(1, 1, kernel_size=[1, 1], stride=(1, 1), bias=False)

with torch.no_grad():
  md.weight.fill_(1.0)

created random input
inp = torch.randn(1, 1, 2, 2)
inp

tensor([[[[-0.5713,  0.0483],
          [ 0.3396,  0.9315]]]])

passed through that layer
ap = md(inp)
ap

tensor([[[[-0.5713,  0.0483],
          [ 0.3396,  0.9315]]]], grad_fn=<MkldnnConvolutionBackward>)

These were same.
There was no effect on passing through that layer.
How can I achieve the same thing when I add it into the model