As far as I understand the best way to optimize a model for CPU a year ago was transfer to Caffe2.
Now with PyTorch 1.0 what is the best option? I describe my situation and thoughts below.
In my case architecture is the next: https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.py 18. I tried to use onnx, but some of the model operations are not supported, such as masked selects clf_h = clf_h[flat == self.clf_token, :] in particular. So I guess it’s not an option for me.
So with PyTorch alone I suppose there are 2 inclusive options: 1) pytorch script (v1.0) and 2) compilation tricks.

To turn the model into pytorch script tracing mechanism must be enough I guess, because I don’t have complex control flow (I haven’t tried yet though)
By compilation tricks I mean pytorch builds with MKLDNN or anything else which can accelerate the inference on CPU (found this stuff as well https://github.com/intel/pytorch#bkm-on-xeon 42)

I install PyTorch 1.0 with pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html. It’s built without MKLDNN, right? I didn’t notice any performance boost just by switching to this version from 0.4.