It's the same line of reasoning that we have stopped referred to, say, people with schizophrenia as schizophrenics or people who have diabetes as diabetics. Instead of referring to people by their traits ("colored" people) we refer to them as people who happen to not be white. It's the difference between "othering" a group based on a trait or including the group who happens to have that trait. You can call it semantics, but word choices matter.

Edit: also not all people of color are black, which is a term that is used *when referencing black communities*. The Asian and Latinx communities have also been known to use these terms in reference to themselves.