I printed your quantized model def before scripting: https://gist.github.com/vkuzo/edb2121a757d5789977935ad56820a24 6
One improvement would be to fuse subsequent Conv-ReLU modules together, so they can use the faster fused quantized kernel:
import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(4, 4, 1),
    nn.ReLU(),
)

# Fuse each Conv and ReLU (implement this for your model)
torch.quantization.fuse_modules(model, [['0', '1']], inplace=True)
print(model)

# prepare
torch.backends.quantized.engine = 'qnnpack'
model.qconfig = torch.quantization.get_default_qconfig('qnnpack')
torch.quantization.prepare(model, inplace=True)

# calibrate (toy example)
input_data = torch.randn(4, 4, 4, 4)
model(input_data)

# convert
torch.quantization.convert(model, inplace=True)

# should see QuantizedConvReLU2d module
print(model)

If you still see a performance gap after this, might be good to check if QNNPACK is enabled on your target device.