I left the default settings for the notebook which I believe has the runtime set to GPU but I’ll double check that setting and rerun anyways. Even so, I was expecting comparable latency running the TFLite model on my macbook (quad core i7, 16gb ram) with 4 threads to the benchmarks listed in the notebook for pixel 4?
I’m going to try build tf from source rather than installing via pip as I know that should improve performance and I believe the benchmarks in the notebook are for the integer quantized model so I’ll also try that.
For the non-integer quantized model, running on TF installed via pip with 4 threads, does 0.6-0.75s per inference sound reasonable or is it likely I’ve messed something up along the way?
FYI the only bit of the notebook I changed is the detect_objects:
import time
start = time.perf_counter()
interpreter.invoke()
end = time.perf_counter()
print(end - start)

Edit: Tried with integer quantized model in colab with similar (slightly worse) results. I also tried running a separate call to invoke before timing in case there was some sort of model loading overhead but that didn’t change anything. Also using that model I get an error IndexError: index 25 is out of bounds for axis 0 with size 25 in the detect_objects method because the output_tensor[3] (which I assume is num_detections) has more items than the score output tensor. Easily fixed but just something I noted.