Why do fighter jets still have guns/cannons? I noticed that the "new" F-35 still has a GAU-22/A cannon installed in it -- albeit with only  182 rounds (more with ammo pods). 

Why do fighters still have cannons? The day of dogfighting is long over, now it's just fire some missiles and forget.  Even if they expended all their missiles, 182 cannon rounds is only 6 seconds of firing  (I think the GAU is rated to 1800 rpm).

The only reason (and I'm not sure if I should answer my own question) is that they need something to shoot warning shots over the bow of another airplane. But that seems an awful amount of weight for this rare occasion. And it'd mean loading tracer rounds.

Or is just a vestigial organ, like the sabres on the side of full dress military uniforms?
The idea that missiles will be all a fighter aircraft needs was prevalent in the late Fifties. The McDonnell-Douglas F-4 Phantom II carried initially only missiles, but at the start of the Vietnam war this turned out to be inadequate. The long-range missiles back then were rather unreliable, and in a real conflict things turn out always different than anticipated. As von Clausewitz said, the plan is the first casualty of war.

F-4s frequently found themselves in close-combat situations for which they were inadequately prepared. Even today, a gun gives the pilot a lot more options, and if the situation is unclear, close-up visual inspection before shooting is still vital to avoid politically embarrassing situations.

From the F-4C on, F-4s were equipped with a gun. First in external pods, from the F-4E as an internal gun. What also helped to make the F-4 more effective was better training for pilots and ground crew, so the missiles were in better shape and better applied.