Monte Carlo dropout and independent dropout masks within mini batches?
One could parallelize MCD inference by having multiple instances of a given item in a mini-batch. However, in order for that to work the dropout masks have to be independent for all the members of a given mini-batch. Normally it would be faster to re-use a mask across members so I’m curious how it’s done in torch. Thanks.
mattmacy
In pytorch dropout masks are independent for all the samples in the minibatch (a single mask the size of the whole input is generated).
ngimel
Good to hear. Thanks.
mattmacy