Really interesting conversation.
Another topic is whether its better to have one critic network per agent or one critic network for each cluster of homogenous agents, as in the centralized training scenario the input vector will be the same for every critic. Some claim that this might have better performance.
However, I suppose that the attention mechanism scenario is can only be applied for multiple critic networks.

You can of course permute the order of states if your agents are homogenous, this  may  increase network robustness.

What I did is concatenating the states and feed them to every critic network with the same order. I could instead try to always feed the critic of agent i, with the state of agent i being in the start of the concatenated states.
Thank you for your detailed answer @iffiX !