Hello @supriyar,
Thanks for your comment!

Are you sure about Sigmoid and Hardsigmoid? Because when I look into models’ print after prepate_qat and I see that the only activations without FakeQuantizer are Sigmoid and Hardsigmoid. While HardSwish has FakeQuantizer attached:

  (2): Hardswish(
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=tensor([1], device='cuda:0', dtype=torch.uint8), observer_enabled=tensor([1], device='cuda:0', dtype=torch.uint8),            quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine, ch_axis=-1,         scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)
    )
  )
........................................................
  (2): Sequential(
    (0): DeQuantStub()
    (1): Sigmoid()
    (2): QuantStub(
      (activation_post_process): FakeQuantize(
        fake_quant_enabled=tensor([1], device='cuda:0', dtype=torch.uint8), observer_enabled=tensor([1], device='cuda:0', dtype=torch.uint8),            quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine, ch_axis=-1,         scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
        (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)
      )
    )
  )


Just did it and it didn’t help.
Already seen that, thanks, but nothing new in terms of code for me 

Does it take longer in terms of epochs? Because timing for sure is increased.
Ok, that’s how I do that.

I am using Catalyst framework, so what I am doing is that I am preparing model for QAT once stage is started. Here is my code:
class QuantizationAwareTrainingCallback(Callback):
    def __init__(
        self,
        backend: str = "fbgemm",
    ):
        super().__init__(order=CallbackOrder.Internal + 1)

        assert backend in ["fbgemm", "qnnpack"], "Unknown backend type"

        self.backend = backend

    def on_stage_start(self, runner: "IRunner") -> None:
        # model is already fused
        runner.model.train()

        torch.backends.quantized.engine = self.backend
        runner.model.qconfig = torch.quantization.get_default_qat_qconfig(self.backend)
        runner.model = torch.quantization.prepare_qat(runner.model, inplace=False)

        runner.model.apply(torch.quantization.enable_observer)
        runner.model.apply(torch.quantization.enable_fake_quant)