Hello !
As thomwolf said, we are currently working on a much performant version of the TF models and then for now, yes, the PyTorch models are more optimized than the TF ones.
Bert-as-a-service works faster because it is highly optimized for inference by making:

BERT using mixed precision (thing that we can do as well)
freeze the model
powerful/scalable service API

If you are looking for a performant inference for your TF model I suggest you to take a look at ONNX, we provide a script 59 to create your own optimized ONNX model in the repo. And afterwards you can run a Triton server 46 to provide an over your model.