It’s actually simpler, to skip the first conv for example, there are two step:
Step 1: Move the quant stub after the first conv in the forward function of the module.
For example in the original quantizable module, quant stub is at the beginning before conv1:
Class QuantizableNet(nn.Module):
    def __init__(self):
        ...
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.conv1(x)
        x = self.maxpool(x)
        x = self.fc(x)
        x = self.dequant(x)
        return x

To skip the quantization of conv1 we can move self.quant() aftert conv1:
Class QuantizableNet(nn.Module):
    def __init__(self):
        ...
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.conv1(x)
        x = self.quant(x)
        x = self.maxpool(x)
        x = self.fc(x)
        x = self.dequant(x)
        return x

Step 2: Then we need to set the qconfig of conv1 to None after prepare(), this way PyTorch knows we want to keep conv1 as float module and won’t swap it with quantized module:
model = QuantizableNet()
...
torch.quantization.prepare(model)
model.conv1.qconfig = None