Managed to get it to look correct (from looking at print(quant_model)) and not error out by only preparing qat on backbone0, and inserting the Quant/DeQuantStub into the nn.Sequential itself
...
    self.backbone0 = nn.Sequential(
            QuantStub(),
            nn.Conv2d(3, 1, 1, bias=False),
            nn.BatchNorm2d(1),
            nn.ReLU(),
            DeQuantStub()
        )
...
model.backbone0.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
torch.quantization.prepare_qat(model.backbone0, inplace=True)

I notice that when I forward zeros through it, it has a random chance of having either the same output as the non quant-converted model or drastically different outputs.
Is this the wrong way to get backbone0 quantised?