@supriyar I have tried to load a normal trained fp32 model, and continue with QAT. I disable observer after 2 epoches, and freeze bn after 2 more epoches.
When I trained with single GPU, the loss was around 13(slightly increase after disabling observer). Everything goes smoothly.
When I trained with 2 GPUs, the loss increased to ~1200 after disabling observer. More iterations seems useless.
I further printed out the first layer during QAT. I found that quant parameters was not updated(compared to using single GPU). Scale and zero_point stuck at 1 and 0, and observer’s min_val & max_val keep empty.
Part of the output is as follows:
Sequential(
  (conv): ConvBnReLU2d(
    3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
    (weight_fake_quant): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
  )
  (bn): Identity()
  (act): Identity()
)
lr: 0.000002    epoch: 0/80     step: 13        train_loss: 13.425(xy: 2.847, conf: 6.531, cls: 4.047)   Sequential(
  (conv): ConvBnReLU2d(
    3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
    (weight_fake_quant): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
  )
  (bn): Identity()
  (act): Identity()
)
lr: 0.000004    epoch: 0/80     step: 26        train_loss: 13.825(xy: 2.980, conf: 6.424, cls: 4.421)   Sequential(
  (conv): ConvBnReLU2d(
    3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
    (weight_fake_quant): FakeQuantize(
      fake_quant_enabled=True, observer_enabled=True,            scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0')
      (activation_post_process): MovingAverageMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
    )
  )
  (bn): Identity()
  (act): Identity()
)
lr: 0.000006    epoch: 0/80     step: 39        train_loss: 13.173(xy: 3.110, conf: 6.385, cls: 3.678)