I see, if this happens for conv layers that have fusion with batch_norm then the possible reason is here pytorch/conv_fused.py at master · pytorch/pytorch · GitHub
where we set the biases to zero and add the original as part of the batch_norm operator.