I suppose that you want to run this on a Raspberry.
Have you explored the option to use an accelerator like this one:

  
      

      Coral
  

  
    

Get started with the USB Accelerator | Coral 9

  Learn how to set up the Coral USB Accelerator and run some demo code.


  

  
    
    
  

  


Here you can find the performance of some reference models for the object detection on Google Coral TPU:

  
      

      Coral
  

  
    

Models - Object Detection | Coral 9

  Models that identify multiple objects and provide their location.