Hi Martin,
This is a very hard question to answer. There’re many BERT models with very different sizes and complexity and this affects how fast the model can do inference. Also the machine in which you are running inference impacts the time. Do you have a gpu to help? Which one?
So it’s not easy to answer that.
What I’d do to try to optimize the process is try the newest version of TF and BERT models.
You can try a more modern BERT model following this guide: Making BERT Easier with Preprocessing Models From TensorFlow Hub — The TensorFlow Blog 2
if the timing is to high, you can try a smaller BERT and still keep a high accuracy.