I noticed that Quantization of MobileNet-V2 model was working fine for me, just like in the pytorch tutorial https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html?highlight=static%20quantization.
So, I started debugging by comparing with Mobilenet-V2. I noticed two changes, that helped my model to also achieve better performance.

using _make_divisible(v, divisor, min_value=None) function(in the MobileNet-V2 code) or manually setting the number of channels to be divisible by 8.
use padding = (kernel_size - 1) // 2 in convolution layers, instead of  padding=0.

I changed the model to following:
class ConvBNReLU(nn.Sequential):
    def __init__(self, in_planes, out_planes, kernel_size=3, stride=1, groups=1):
        padding = (kernel_size - 1) // 2
        #padding = 0
        super(ConvBNReLU, self).__init__(
            nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groups=groups, bias=True),
            #nn.BatchNorm2d(out_planes, momentum=0.1),
            # Replace with ReLU
            nn.ReLU(inplace=False)
        )
    def fuse(self):
        fuse_modules(self, ['0', '1'], inplace=True)
    
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        block = [
            ConvBNReLU(1,8,3,2),
            ConvBNReLU(8,16,3,1,8),
            nn.Conv2d(16,32,1,1),
            ConvBNReLU(32,8,1,1),
        ]
        self.features = nn.Sequential(*block)
        self.classifier = nn.Linear(8,2)
        self.quant = QuantStub()
        self.dequant = DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.features(x)
        x = x.mean([2, 3])
        x = self.classifier(x)
        x = self.dequant(x)
        return x

and achieved ~38MB reduction in memory in Android by using Quantization. by using the above 2 points.