And many people disagree with you and don't find it to be "as simple as that".  Who gets convicted depends on who gets arrested, and much has been written about the disparate rates of stops and arrests and the disparate ways in which plea bargaining often occurs.
Michelle Alexander's work gives many statistics and support for the positions taken in her book.  Her position is diametrically opposed to yours.