Does dropout have any benefits when overfitting isn't a concern? I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training dataset to overfit on. I've confirmed experimentally that performance on a fixed validation dataset is no worse than training performance.
Is there any potential/likely benefit to including dropout in this model? Is there anything to support it resulting in better trained performance when overfitting isn't a concern?
I've tried some basic experiments and found that dropout leads to dramatically reduced performance in the early stages of training, but I haven't let it run for very long.

Even if overfitting is not a concern, dropout could still help. Neural networks are (most of the time) trained with backpropagation, i.e. stochastic gradient descent (SGD). And it is known that the stochastic aspect of SGD often helps it to escape local minima. While the main source of stochasticity is the random blocks in each epoch (with the blocks much smaller than the total dataset), techniques like dropout (might) add a different type of stochasticity that could perhaps improve your results.
So, while you can never be certain of anything with DNNs, I would definitely give dropout a try.
