Hi @olaffson , I would like to add that batching on inference is often detrimental (on real loads, at least in PyTorch, less experience with TF). The reason is the padding of tokens.
The real game changer for GPU inference, is to do the processing (tokenizer) on a different thread than the inference on GPU, to keep it busy 100%. In PT land, it’s done with DataLoader, and I guess that’s tf.Dataset’s purpose too.