Maybe peaceful aliens don't come here in mass because they know better already. Maybe natural disease and viruses on habitable planets itself keeps aliens from coming to "living worlds" because the cross transmission of disease is inevitable over time and both species will die from it on said planet. So everyone keeps to themselves except at designated spots where small groups could meet for a time. Core worlds for people who aren't violent all the time and who've been in the interstellar travel club for a while. You know something like that. Now that wouldn't apply to bad ones obviously, they would just sterilize the entire planet and then harvest whatever they came for.