Ideally, a suitable value for num_workers is the minimum value which will give batch loading time <= inference time. This way, when our model is working on inference of previous batch, data-loader would be able to finish reading the next batch in the mean time.
However, the maximum number of num_workers is also dependent on available cpu resources, so we might not always be able to achieve that ideal number of num_workers.