For reference I try to implement this paper: https://openreview.net/pdf?id=HkO-PCmYl 29 But for helping me you should not have to read it.
This is my forward pass in my nn.Module:
    def forward(self, x):
        if self.training:
            alpha = Variable(torch.rand(x.size()[0]).cuda())
        else:
            alpha = Variable(torch.FloatTensor([0.5]).cuda())

        p1 = self.path1(x)
        p2 = self.path2(x)

        alpha = torch.unsqueeze(alpha, dim=1)
        alpha = torch.unsqueeze(alpha, dim=2)
        alpha = torch.unsqueeze(alpha, dim=3)

        alpha = alpha.expand(p1.size())

        x = alpha * p1 + (1 - alpha) * p2
        return x

So two questions:

Is there a better way to handle alpha? All the torch.unsqueeze look ugly.
In the backward pass I want to handle the gradient with alpha=0.5 instead of the random alpha from the forward pass. How would I implement this? Maybe you can give a working implementation? I saw http://pytorch.org/docs/master/autograd.html#torch.autograd.Function 4, but I am not sure how to use it…