Multiply feature map by a learnable scalar
Alexey_Chernyavskiy
<SCODE>matrix = Variable(torch.rand(3,3))
scalar = Variable(torch.rand(1), requires_grad=True)
output = matrix * scalar.expand_as(matrix)
<ECODE>
fmassa
in my Model definition: <SCODE>    self.multip = torch.autograd.Variable(torch.rand(1).cuda(), requires_grad=True)
    self.multip = self.multip.cuda()
<ECODE> in my Model forward: <SCODE>def forward(self, x):
  x1 = self.relu(self.conv1(x))
  x2 = self.relu(self.conv2(x))
  x1 = x1 * self.multip.expand_as(x1) # multiply x1 output by learnable parameter "multip"
  x = torch.add(x1, x2)
  return x
<ECODE>
Alexey_Chernyavskiy
Quick example: <SCODE>class Model1(nn.Module):
    def __init__(self):
        super(Model1, self).__init__()
        self.multp = Variable(torch.rand(1), requires_grad=True)

class Model2(nn.Module):
    def __init__(self):
        super(Model2, self).__init__()
        self.multp = nn.Parameter(torch.rand(1)) # requires_grad is True by default for Parameter

m1 = Model1()
m2 = Model2()

print('m1', list(m1.parameters()))
print('m2', list(m2.parameters()))
<ECODE>
fmassa
Now, how can I specify a special learning rate for this new param? I used to do something like this for my convolutional layers to make a specific smaller learning rate for layer ‘conv2’. UPD: I replaced my ‘custom’ Adam (see above) that had different learning rates by a regular Adam which has the same learning rate for all the layers, and it worked - my multp started to become updated during the course of training (it also worked with SGD). However my question with setting a special learning rate for the nn.Parameter remains.
Alexey_Chernyavskiy
You can simply do something like <SCODE>optimizer = optim.Adam([{'params':[model.multp], 'lr':opt.lr*0.1}], lr=opt.lr)
<ECODE>
fmassa
Great, it worked! Thank you.
Alexey_Chernyavskiy
Hey, I was wondering if is possible to mutiply each feature map by a different scalar?
Andrefmds
Maybe using a 1x1 conv with group=n_channels?
Chenhongyi_Yang
Hi did you work it out? I also need to do something similar.
mbutt