Question: What constitutes a vampire "nest"? Is it just the home of several vampires, or does it have to actually be "nest" like (i.e. dirty, dark, stinky, unpleasant)? 

For instance, I've never seen the Cullen's house referred to as a nest, since they're sort of civilized. But in the first season of True Blood there were three vampires living in a house and it was referred to as a nest. Is there a distinction? Or just the preference of the author writing the vampires? Is Jean Claude's building in the Anita Blake series considered a nest? 

Also, what is the first mention of a 'vampire nest' in fiction?

Answer: 
It depends on the series that is under consideration.

Typically, a 'nest' of vampires is used by mortals who are exterminating vampires - Blade, Buffy, etc will use the term. This plays into their human desire to distance their foes from humanity.

In works where vampires are portrayed more sympathetically, every effort is made to 'humanize' the vampires (or make them 'better than human').

I'm unable to find a reference to the first usage of 'nest' to refer to a group of vampires. The earliest I can recall is Blade (or possibly John Carpenter's Vampires) but both of those are fairly recent (in terms of movies).
the word has different connotations in different works mainly because there are different kinds of vampires

john carpenters vampires had a "queen" vampire that had a psychic link to his children (its been awhile but I think the sired vampires had a basic drone level of intelligence as well)

in true blood Nests are described by Bill as causing the vampires living in them to become more cruel the behavior could be mystical or it could just be behavioral reinforcement 

buffy and angel both referenced nests but I dont think they meant anything more than a place where a group of vampires lived (unlived) together. they also used the word nest to refer to things other than vampires

personally I always kinda assumed the word nest was the proper way to refer to a group of vampires, kinda like a murder of crows or a waddle of penguins

as to the first usage I cant help you