Thanks a lot for your kind and informative reply, @wassimseif !
When I tried exclusively quantization the mAP also drops enormously and I wondered whether I was doing it correctly. Will try calibration too - thanks for the suggestion!
Just to make sure though: At the end of the forward pass, there are two dequant stubs added then, one for the locations and one for the confidences? Initially I thought that I should dequantize after the base net  because I suspected that such find grained localization cannot be done well with 8bits but requires the more expressive 32bit representation.