lucastononrodrigues:

self.B =self.reset_parameters(nn.Parameter(torch.Tensor(self.projection_dim,10),requires_grad=True)).unsqueeze(0).repeat(num_heads,1,1)


Not sure what is the problem extractly, but try to do this:
self.B = nn.Parameter(torch.Tensor(self.projection_dim,10))
(The way you use reset_parameters is wierd, I don’t even know why it works. You should just run self.reset_parameters at the end of init)
in the forward use,
self.B.unsqueeze(0).repeat(num_heads,1,1)