Dropout for RNNs
I wonder if there would be an elegant way to use the same dropout mask on sequences for RNNs, or it would be better to implement a module. (Dropout option in the current RNN module just regard the entire sequence output as a single output. Right?) Thanks!
supakjk
You could write your own module, where you process the whole sequence, sample the mask once at the beginning and just do an element-wise multiplication after each step. Here’s a gist: <SCODE>def forward(self, x):
    mask = Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli_())
    hidden = None
    outputs = []
    for t in x.size(1):
        output, hidden = self.rnn(x[:, t], hidden)
        outputs.append(output * mask)
    return outputs, hidden
<ECODE>
apaszke
Regarding the current RNN modules, they use a different mask for each time step.
apaszke
Performance-wise, running rnn on the whole input sequence, expanding mask and applying it to the whole rnn output will probably be better than having a loop over time.
ngimel
emanjavacas
apaszke
emanjavacas
Yes, if a separate mask for each timestep is ok, then you can just use the built in modules. They will be the fastest.
apaszke
<SCODE>def _make_noise:
	return input.new().resize_(1, input.size(1), input.size(2))'<ECODE>
supakjk
apaszke
What I did to use the same dropout mask for different time steps was inheriting classes as follows: <SCODE>class SeqConstDropoutFunc(torch.nn._functions.dropout.Dropout):
        def __init__(self, p=0.5, train=False, inplace=False):
                super(SeqConstDropoutFunc, self).__init__(p, train, inplace)

        def _make_noise(self, input):   # for timesteps X batches X dims inputs, let each time step has the same dropout mask
                return input.new().resize_(1, input.size(1), input.size(2))

class SeqConstDropout(nn.Dropout):
        def __init__(self, p=0.5, inplace=False):
                super(SeqConstDropout, self).__init__(p, inplace)

        def forward(self, input):
                return SeqConstDropoutFunc(self.p, self.training, self.inplace)(input)
<ECODE> Thanks.
supakjk
Yes I think it’s best to reimplement it yourself. It’s a very simple function.
apaszke
I think you should sample the bernoulli using: <SCODE>Variable(torch.bernoulli(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout)))
<ECODE>
seba-1511
Is there an elegant implementation of it already? I guess the idea of using the same dropout mask is from “A Theoretically Grounded Application of Dropout in Recurrent Neural Networks”.
cosmozhang1988
Did you get it implemented in pytorch already?
cosmozhang1988
Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli_()) is not correct. The correct form is Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli())
026f9ee49b7010c38742