I made some changes in  the forward and the model runs. Basically, I need the main diagonal of a matrix multiplication, so I customized a   col_wise_mul function.
Before:
def col_wise_mul(m1, m2):
    result = torch.zeros(0).to('cpu')
    for i in range(m1.shape[1]):
        v1 = m1[:, i, :]
        v2 = m2[:, i]
        v = torch.matmul(v1, v2).unsqueeze(1)
        result = torch.cat((result, v), dim=1)
    return result

x = element_wise_mul(x_feature, label_feature)

Now:
 x = torch.diagonal(torch.matmul(x_feature, label_feature), offset=0).transpose(0, 1)

I am wondering why the customized function does not work? Is there any other ways to accomplish the goal as I have two very large matrixes (400, 30000). If I do multiplication first and then get the diagonal, it cost too much memory.