It's both. There are people who truly think they're superior to you just because they're not virgins, without any legitimate knowledge other than that to judge you by, but more often "virgin" is just a casual insult. Pretty much exactly how people used to use "nerd" as an insult. You had the airhead jock who meant it and the friend who honestly didn't care.