I have tried:

permute order randomly
make agent i first
order states from 1 to n
In my undergraduate thesis, there is no significant difference.

A visualization of attention weight, using method 3:
image1063×454 33.8 KB
For single/multiple critics in the homogenous scenario:
I have tried:

one central critic
one critic for every agent
one critic for every agent, but average their parameters every few episodes. ( by mean() their state dict and broadcast new mean parameters to all critics)
1 performs best and 3 performs the worst, I assume that averaging has damaged the information collected by critic.

You may try these methods yourself as I have only test my solution with one multi-agent environment modified from the bipedalwalker-v2, contact me if you would like to take a look at the modified implementation: