Thank @fmassa, I have another question though.
How do I perform training with such a multiplier?
Right now, I have this:
in my Model definition:
    self.multip = torch.autograd.Variable(torch.rand(1).cuda(), requires_grad=True)
    self.multip = self.multip.cuda()

in my Model forward:
def forward(self, x):
  x1 = self.relu(self.conv1(x))
  x2 = self.relu(self.conv2(x))
  x1 = x1 * self.multip.expand_as(x1) # multiply x1 output by learnable parameter "multip"
  x = torch.add(x1, x2)
  return x

and in my training code:
…
optimizer.zero_grad()
loss = criterion(model(input), target)
loss.backward()
optimizer.step()
As you see, nothing out of the ordinary. But when I print the value of model.multip.data[0], I see that the initial value of multip remains unchanged (all other params in my Model do change). I deduced that probably I messed somewhere and did not apply the gradients to self.multip
Am I right?