Igusm, thanks for the info. I do have GPU on servers, but in development I prefer to running models on my Mac with CPU only for a small set of data. Do you have suggestions on how to use cpu/gpu in development stage?
Another thing I noticed related to my question above is that, when I run the models on the same machine with the same data,  the time could be significantly different. On one run, each prediction may take 30 or 40 ms on average, and another run it may take 20 ms. Also, for similar text length, one prediction’s time differ quite a lot from another prediction. It seems the prediction’s times are unstable. Is this normal?