I noticed two things so far:
Pytorch has issues with branches in the model for some reason. that is, lets consider SimpleNetwork here.

class SimpleNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(10)
        self.relu1 = nn.ReLU()

        self.prelu_q = PReLU_Quantized(nn.PReLU())
        self.bn = nn.BatchNorm2d(10)

        self.prelu_q_or_relu = torch.relu if use_relu else self.prelu_q
        self.bn_or_identity = nn.Identity() if disable_single_bns else self.bn    

        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()
    
    def forward(self, x):
        x = self.quant(x)

        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu1(x)

        x = self.prelu_q_or_relu(x)
        x = self.bn_or_identity(x)

        x = self.dequant(x)
        return x

This by default results in the infamous error stated in the op. However, if I simply remove :
self.bn_or_identity = nn.Identity() if disable_single_bns else self.bn    

and simply use the
self.bn = nn.BatchNorm2d(10)

in the forward pass, I no longer see that error!.
I tried to do the same thing to ResNet18, and it seems, all previous bns are fine except the bn3 in the ResNet model (the penultimate layer) which regardless of what I change, still give that error !