Exponential smoothing is just a recursive weighted mean. But it is not permutation invariant, to model that you should use simple mean(). If first input is special, you’re free to handle is separately.
Self-attention - I don’t understand it well enough to elaborate. Its use for aggregation is described in Attentive Neural Processes paper.