Just for me to understand better, you want to do something like that?
for input, target, ID in dataloader:
  # Assume batch_size == input.size(0)
  features = SuperEncoder()(input) # Is that right? You encode enverything here?
  # We have features.size(0) == batch_size
  avg_features = # Some ops that average all the features with the same ID?
  # But now avg_features.size(0) == ??
  # What shall we do with the targets? Should they be averaged as well?

  # More code