Solved by Vasiliy_Kuznetsov in post #2 
              
              
                Which layers to quantize is controlled by the qconfig, when we do model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm') it applies the defaults settings of which modules to swap to the entire model.