you can set qconfig only for the sub module that you want to quantize. e.g.
class M(torch.nn.Module):
   def __init__(self):
      self.conv1 = ...
      self.conv2 = ...
m  = M()
m.conv1 = qconfig # if you only want to quantize conv1