In the centralized training, decentralized acting scenario, yes, critics will observe state vectors of other agents, and it is usually done by collecting states of all agents and concatenate them into a fixed dimension state tensor and feed into the critic network. You can of course permute the order of states if your agents are homogenous, this may increase network robustness.
It is really unknown whether your network will take agent i more seriously than other agents, however, you may explicitly apply the attention mechanism, which is usually used in NLP areas, to sort of peeking into the thought of your critic network.