You want avg_features.size(0) == batch_size ?
But what should each of these contain? For each sample, its feature should be the average of all the features with this ID? That would mean that many samples will have the same features right?