There are a couple things might be related to the error:

You are using original vgg16, you need some modifications so that it can be quantized. You can take a look at torchvision/models/quantization/resnet.py as well as the tutorial: https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html 56 to see how to modify the model.
QuantStub() need to be added to the model instead of using it directly.
Between prepare() and convert() you need to run the model to collect histogram, which will be used to determine the scale and zero point of the quantized model.