Question: What constitutes a vampire "nest"? Is it just the home of several vampires, or does it have to actually be "nest" like (i.e. dirty, dark, stinky, unpleasant)? 

For instance, I've never seen the Cullen's house referred to as a nest, since they're sort of civilized. But in the first season of True Blood there were three vampires living in a house and it was referred to as a nest. Is there a distinction? Or just the preference of the author writing the vampires? Is Jean Claude's building in the Anita Blake series considered a nest? 

Also, what is the first mention of a 'vampire nest' in fiction?

Answer: 
It depends on the series that is under consideration.

Typically, a 'nest' of vampires is used by mortals who are exterminating vampires - Blade, Buffy, etc will use the term. This plays into their human desire to distance their foes from humanity.

In works where vampires are portrayed more sympathetically, every effort is made to 'humanize' the vampires (or make them 'better than human').

I'm unable to find a reference to the first usage of 'nest' to refer to a group of vampires. The earliest I can recall is Blade (or possibly John Carpenter's Vampires) but both of those are fairly recent (in terms of movies).
The first time I encountered the term was in the movie The Lost Boys (1987).