def forward(self, x):
    if self.training:
        alpha = Variable(torch.rand(x.size(0)).cuda())
    else:
        alpha = Variable(torch.FloatTensor([0.5]).cuda())

    p1 = self.path1(x)
    p2 = self.path2(x)

    alpha = alpha.view(alpha.size(0), 1, 1, 1).expand_as(p1)

    x = alpha * p1 + (1 - alpha) * p2
    return x

For your second question, wouldn’t you need to do another forward pass using 0.5 then? Otherwise the output and intermediate outputs are incorrect to calculate gradients.