Well my problem just sort of got solved… I managed to get my hands temporarily on a GTX 1080 Ti and the difference in perfs are actually that big: 10ms on average for all resnet50. I can’t say I expected that.
edit: Actually it seems to be a matter of configuration… if I run on Linux on my own machine, I get 12ms, which is much better (vs 58ms on Windows). Guess I gotta keep exploring…