Ardeal:

what is your machine configuration?
what is your test data? for example, what is the time needed for each iteration/batch?


I don’t work with this task. And always use just google colab. My explanation is just how I understand the topic from reading the documentation, other discussions and my own experience.
As for debugging your issue, I would suggest trying to check your data pipeline to understand why is it so memory hungry. It might be there is an opportunity to optimize it. Also, It seems like 2 workers are actually enough for the task, as you can see that overall training time doesn’t improve much with further increasing number of workers.