unfortunately, I do not understand what do you mean by saying “manually or through graph quantization”
If you mean, do I profile Torchscript model or original python one: I’ve done both, and both of them show bad performance. The layer-by-layer profiling via torchprof does not work with TorchScript models, but I expect that “slow” layers stay slow both in original model and in a Torchscript one, even if timing original model is not so precise
Quantization is done the same way as in MobileNet tutorial 3.
Here is the code snippet:
def try_config(qconfig, one_thread_inference=True, calibration_max_batches=None, metrics_max_batches=None):
    # Fuse modules  (my own implementation for EfficientNet; do you need it?)
    q_model = fuse_modules()
    
    # apply config (this looks complex because I do not quantize _conv_stem (1st convolution)
    for block in q_model.feature_extractor._blocks:
        block.qconfig = qconfig    
    q_model.quant = QuantStub(qconfig)
    print(qconfig)

    torch.quantization.prepare(q_model, inplace=True,
                               white_list=(
                                   torch,nn.Conv2d,
                                   torch.nn.BatchNorm2d,
                                   torch.quantization.stubs.QuantStub,
                                   torch.nn.quantized.modules.functional_modules.FloatFunctional
                               ))
    q_model.eval()
    
    print('Post Training Quantization Prepare: Inserting Observers')
    if one_thread_inference:
        torch.set_num_threads(multiprocessing.cpu_count())
    inference(q_model, dev_loader, max_batches=calibration_max_batches)  # custom func just for model inference

    print('Post Training Quantization: Calibration done')

    # Convert to quantized model
    torch.quantization.convert(q_model, inplace=True)
    print('Post Training Quantization: Convert done')