I am creating a layer which will acts as both embedding as well as linear layer. This is to tie the weights.
class embedding_linear(nn.Module):
    def  __init__(self,vocab_size, dmodel=dmodel, pad=True):
        '''
        Tied weights for decoder embedding layer and pre-softmax linear layer.
        
        vocab_size: size of vocabulary used. It may be different for both source and target
        dmodel: dimension of the word vector
        pad: the pad index in the vocabulary
        '''
        super(embedding_linear,self).__init__()
        self.dmodel = dmodel
        self.weights = nn.Parameter(torch.Tensor(vocab_size,dmodel))
        self.bias = nn.Parameter(torch.Tensor(dmodel))
        self.weights.data.normal_(-1,1)
        if pad:
            self.pad_idx = 0
            self.weights.data[0].fill_(0)
        else:
            self.pad_idx = -1
        
        
    def forward(self, inputs, emb=True):
        if emb:
            outputs = F.embedding(inputs, self.weights * (self.dmodel ** 0.5), self.pad_idx, False,2, False, False)
        else:
            outputs = F.linear(inputs,self.weights.t(),self.bias)
        return outputs
            

But when I test the layer with the following command it always returns zeroes
both negative and positive zeroes

image.png1100×185 15.7 KB

initially I thought it must be precision problem and tried casting the outputs to other types by using the .type(), But nothing changed.
I then tried using the official embedding layer and got this output

image.png1090×167 18.4 KB

where it was not all zeroes.
I also matched my implemention from the official repo and they were almost same.
Please tell me what to do? and why is it returning all zeroes.