Thanks, but thats not a solution to me, removing bn drastically affects the performance and aside from that, @supriyar says Pytorch has a quantized version of BatchNorm in place and it should have got converted in first place!
So its not known what is missing or what else needs to be done.