This is working but I am not sure if it is correct, as it doesn’t have my expected behavior.
mode_val = weights.mode(dim=0)[0]
mode_val = mode_val.unsqueeze(0).repeat_interleave(2, dim=0)
(weights * (weights != mode_val)).sum(dim=0)