@mohit7 did you modify the fuse_model function to work with your model (https://github.com/pytorch/vision/blob/master/torchvision/models/quantization/resnet.py#L45 16)?
Currently fusion is only supported for conv + bn or conv + bn + relu. Does your model have a use case other than that?