Hi,
In my model , some parameters’s gradient keeps as zero,  which confused me a lot.  Some necessary information list as below:
The main code about the parameters of zero-gradient:
###
some code
###
# about attention
alpha = F.softmax((bag_embs * self.att_a).mm(self.query_vec), 1)
bag_embs = (bag_embs * alpha).sum(0)
batch_fea.append(bag_embs)
###
some code
###

in the model definition:
self.bias_d = nn.Parameter(torch.randn(3,5))
self.query_vec = nn.Parameter(torch.randn(3,1)
self.att_a = nn.Parameter(torch.randn(3)


in the model weight init:
nn.init.xavier_uniform(self.query_vec)
nn.init.uniform(self.att_a)

And I use the Tensorboad to visualize the model parameters and its gradient:

image.png957×605 42 KB


image.png920×301 12.3 KB

The gradient of att_a and query_vec is always zero. while others such as bias_d is ok.
So What may cause that problem?
Thanks