We have usually quantized the backbone part for detection models while leaving the rest in fp32 and gotten good speedups. For the other part, @Zafar has tried quantizing but the accuracy is usually bad.