The problem was that the classes(in the previous example was 3518) this big number of classes conclude to very small number after the softmax pass. So every time the sum of the previous softmax outputs was all the time bigger than the current output. This led to taking all the time as min the element of the current softmax output concluding in just the sum of all the elements of the current softmax output, which every time is 1.
To change this we have to take the mean of all the previous sums of softmax output.
overall_weights = ((torch.sum(activation[:time+1,:],dim=0))/(time+1)).to(device)