We improved Tensorflow Serving performance by over 70%
FWIW, I believe that the current state of the art for batch-size 1, fp32 inference for ResNet-50 on Intel CPUs is AWS's work in <a href="https://arxiv.org/abs/1809.02697" rel="nofollow">https://arxiv.org/abs/1809.02697</a>. After the low-hanging fruit outside of model execution are picked, this kind of work is probably quite relevant.