KFrank:

What is your intuition about why (and when) dropout works?


I’m a Bayesian at heart, so to me Gal and Ghahramani 2 interpretation seems to be a thing.
In a later paper they show that for RNNs it makes sense to keep the dropout mask fixed between the timesteps to use this interpretation and Gal’s theses has a lot of things on the theme (including “why can we not optimize the dropout probability somewhere, I think”). More generally, it would seem that the hypothetical variational model must make sense as well as the approximation.
What is your take? I always love to hear your thoughts on the more theory-grounded topics.
Best regards
Thomas