Can you point us to what kind of network are you using, is it a CNN like ResNet for example? Also, do you use batches > 1 during inference? So that we can have a better idea on what are your use-cases and what is slow compared to TF.