How to normalize embedding vectors?
Hi, I am using a network to embed some entity into vector space. As the length of the vector decrease during the training. I want to normalize it’s length to 1 in the end of each step. Is there any tool that I can use to normalize the embedding vectors?
maplewizard
I think the best thing you can do is to save the embedded indices, and normalize their rows manually after the update (just index_select them, compute row-wise norm, divice, index_copy back into weights). We only support automatic max norm clipping.
apaszke
If you want to normalize a vector as a part of a model, this should do it: assume q is the tensor to be L2 normalized, along dim 1
samarth-robo
Can I use it to normalise the embedding after each update in the training ?
asdass
samarth-robo
I see, thanks.  But I need to do the constraints on the norms of embeddings( not bigger than 1 ). Do you have any better suggestions to do it?
asdass
<SCODE># suppose x is a Variable of size [4, 16], 4 is batch_size, 16 is feature dimension
x = Variable(torch.rand(4, 16), requires_grad=True)
norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm.expand_as(x))
<ECODE>
jdhao
<SCODE>norm = x.norm(p=2, dim=1, keepdim=True)
x_normalized = x.div(norm)
<ECODE>
moi90
<SCODE>import torch.nn.functional as F
x = F.normalize(x, p=2, dim=1)
<ECODE>
jdhao
Liang
Then in this case I get the following error for any of the above options I try: TypeError: cannot assign ‘torch.autograd.variable.Variable’ as parameter ‘W’ (torch.nn.Parameter or None expected) <SCODE>class myUnit(nn.Module):
    def __init__(self,myParameter):
        super(myUnit, self).__init__()
        self.myParameter = Parameter(myParameter,requires_grad=True)
    def forward(self,input):
        """
        Whatever operation. Just an example:
        """
        self.myParameter = F.normalize(self.myParameter,p=2,dim=1)
        output = self.myParameter * input - 1
        return output<ECODE>
sssohrab
May be you can try: <SCODE>self.myParameter.weight.data = F.normalize(self.myParameter.weight.data, p=2, dim=1)
<ECODE>
Guohai93
Ritwick_Chaudhry
f10w
Your issue seems to be unrelated to the linked post, as your error points towards an error running a DDP model.
ptrblck
Sorry I copied the wrong error message. The above message was for the following code: <SCODE>self.embeddings = torch.nn.Parameter(F.normalize(self.embeddings, p=2, dim=-1))
<ECODE> What I would like to do is to constraint the weights of my model to always lie on a sphere. And I finally found the solution: <SCODE>self.embeddings.data.copy_(torch.nn.Parameter(F.normalize(self.embeddings.data, p=2, dim=-1)))
<ECODE> Hope that this will be useful for future readers.
f10w