Hi @karllessard
Back in TensorFlow 1.12.x I did a benchmark via custom build of TensorFlow with mkl and opt flags. To build the Java part I just followed these instructions: tensorflow/README.md at master · tensorflow/tensorflow · GitHub 8
In my benchmark (training NER) the Intel Cascade Lake w/MKL was close and sometimes better than GPU (using the system’s memory it could have a larger batch size).
That’s being said, I’ve never tried testing the inference. But the training was much faster than a native CPU build on newer CPU architectures.