Does dropout have any benefits when overfitting isn't a concern? I'm training a transformer based deep learning model in a regime where overfitting isn't a concern. Infinite training samples are generated on demand and never repeated, so there is no training dataset to overfit on. I've confirmed experimentally that performance on a fixed validation dataset is no worse than training performance.
Is there any potential/likely benefit to including dropout in this model? Is there anything to support it resulting in better trained performance when overfitting isn't a concern?
I've tried some basic experiments and found that dropout leads to dramatically reduced performance in the early stages of training, but I haven't let it run for very long.

One interpretation of dropout is that you can analog it as an ensemble of many neural nets.
When training, each node in the networks has some probability to be dropped.
This means that in every training step, the network architectures are different.
And when testing, we weightedl combined many networks together into a single one.
However, in training, the dropped networks' capabiliteis are less capatible then the full networks.
Thus the performance of the ensembled one might still not outperforme the full one without dropout in training.
