I’m forced to use p.data, because if I replace p.data with p, the weights won’t be modified. For example, if you print p values before and after the dropout line, you will see that only p.data method works (zeros out some weights). Unless you mean something else?
Good point about moving weight restore after backward call! Thank you.
I have to multiply by 1 - drop_prob because dropout scales its input internally by 1 / (1 - drop_prob), and if I don’t do this the accuracy drops sharply: with drop_prob=0.05 it does not even converge if I don’t scale back the weights. I’m not sure what’s going on, but I suspect it might have something to do with batchnorm. Any ideas?
What do you mean “is it still straightforward?” With my method it’s much easier to apply dropconnect selectively, I don’t have to create wrappers for every single layer type, and I don’t have to modify my model forward function. I agree with your point about safety.