Seems like you only need matrix multiplication here, Weights @ P = averages. Weights of shape (batch, num_embeddings) should replace your d dict and contain “contributions”, that are either zero or 1/emb_count, to get masked averages.