batch_size: use the largest batch size that fits in GPU memory. Scale learning rate and warmup according to the method of “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.” 2
num_workers: how do you tune a Winnebago to get a good lap time at Le Mans? Step 1: replace it with a Ferrari. Step 2: tune the Ferrari.