Question: What constitutes a vampire "nest"? Is it just the home of several vampires, or does it have to actually be "nest" like (i.e. dirty, dark, stinky, unpleasant)? 

For instance, I've never seen the Cullen's house referred to as a nest, since they're sort of civilized. But in the first season of True Blood there were three vampires living in a house and it was referred to as a nest. Is there a distinction? Or just the preference of the author writing the vampires? Is Jean Claude's building in the Anita Blake series considered a nest? 

Also, what is the first mention of a 'vampire nest' in fiction?

Answer: 
I think it is similar to a wasps' nest -- not something cozy but rather like what animals would build. For example, in the Fright Night remake (and maybe in the original, I don't recall) the head vampire simple stores his brood in the basement -- it is a practical place for them, out of the light but they have no comforts nor do they need them -- presumably the head vampire just throws victims to them occasionally and then they emerge from holes in the wall or floor.
The first time I encountered the term was in the movie The Lost Boys (1987).