I have found some very unusual behavior when comparing time of inference run for ResNet50 in image classifier for various backends. I’d very much appreciate any comment why such thing might have happened.
My line looks like this:
./bin/image-classifier <testImages> -use-imagenet-normalization -image_mode=0to1 -m=resnet50 -model_input_name=gpu_0/data -<cpu|interpreter|opencl> -time

Where <testImages> is directory with 1/16/32/64 images depending on the tested batch size. To have a bit clearer picture I have used measured time to calculate images/second value.
And now the strange part…
For OCL I can see logarithmic increase, but for all other backends value is constant. It completely doesn’t matter how big batch size I throw at it, images/second value is the same (+/- small noise).
On generated dot file everything looks reasonable, that is batch size is propagated.
Have you also experienced similar behavior? Or maybe I’m just doing my measurements wrong?