This is hard to answer with only this information.
Are you running the model on a phone? is it using GPU? are you using multi-thread?
Usually what I did in the past to find out why a model is slow on my app was to run the benchmark tool: Performance measurement  |  TensorFlow Lite 1
one thing that could help is decreasing the image input size. 640x640 is quite big and object detection is a complex task by itself.