I’m trying to manually normalize my embeddings with their L2-norms instead of using pytorch max_norm (as max_norm seems to have some bugs). I’m following this link and below is my code:
emb = torch.nn.Embedding(4, 2)
norms = torch.norm(emb.weight, p=2, dim=1).detach()
emb.weight = emb.weight.div(norms.expand_as(emb.weight))

But I’m getting the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/torch/autograd/variable.py", line 725, in expand_as
return Expand.apply(self, (tensor.size(),))
File "/usr/local/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward
result = i.expand(*new_size)
RuntimeError: The expanded size of the tensor (2) must match the existing size (4) at non-singleton dimension 1. at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:308

When I look at the size of norms, it’s (4L,)
Any idea where I’m going wrong? Thanks!