No, you need to follow the static quantization flow, and replace all occurances of the addition with a layer FloatFunctional. For example, if you have a model that looks like that:
class Foo(nn.Module):
  def __init__(self):
    super().__init__()
  def forward(self, x):
    y = x + x
    z = y + 2 * x
    return x

it is not “quantizable”. The reason is that the additions and multiplication are in the forward, and don’t store the state of the scale and zero_point needed for the quantization.
Your quantizable model will looks something like that:
class Foo(nn.Module):
  def __init__(self):
    super().__init__()
    self.first_functional = nn.quantized.FloatFunctional()
    self.second_functional = nn.quantized.FloatFunctional()
    self.third_functional = nn.quantized.FloatFunctional()

  def forward(self, x):
    y = self.first_functional.add(x, x)
    x2 = self.second_functional.mul_static(x, 2)
    z = self.third_functional.add(y, x2)
    return z

Notice that we don’t reuse the float functional modules, and use a different one for every arithmetic operator. This is done so that the quantization parameters for each operation would be computed differently.
Once you rewrite your models, you can run the static quantization (prepare, calibrate, convert steps): Quantization — PyTorch 1.7.1 documentation 15