Question: Computer Science PhD admit here. 

Is it crazy of me to turn down a top school (MIT/Stanford) for a lower-ranked school (think Cornell/UW/Columbia/Michigan) if I think there is better advisor fit?

Obviously all of these are fantastic schools and I am very lucky to be choosing from them. However, I felt like I got along really well with my would-be advisor at the lower-ranked school than I did at MIT/Stanford. People have told me that it's crazy of me to turn down MIT/Stanford since they are, along with Berkeley/CMU, on a different "level" than the other schools.

(My field is somewhat narrow so there is only 1~2 faculty at each of the schools doing research in the area. So regardless of which school I go to, if the advisor doesn't work out, I would be in a dire position.)

EDIT: Both advisors are well known in the field (and both have produced outstanding graduates), but the MIT/Stanford professor is good deal more senior. Also, talking with their current students, the MIT/Stanford professor is known to be pushy and have a "strong" personality (which may work for some people, but not sure whether it will work for me, as I've never worked with such people before).

Answer: 
All of the schools that you are discussing are first-rank computer science schools with global leadership in their areas of specialty. In this area at least, the only real difference between the "top" and "top of the top" schools that you are considering is the *number* of different areas in which the school is a global leader. Thus, if you already have a clear, narrow focus that you know that you want to work in, and there are 1-2 people in that area at all of the schools you are choosing between, the distinction between "top" and "top of the top" is pretty much null. 

Furthermore, your graduate career will be primarily determined by your advisor relationship rather than the school within which it will occur. Beware of being misled by first impressions, but a good match with your Ph.D. advisor is worth much more than the incremental difference between schools.
All of the above answers are correct. However, do trust your first impressions. I didn't click with my first prospective advisor and picked someone else. That prospective advisor failed to get tenure after his next couple of graduate students opted to switch advisors and start new research projects in the 3-4th year of their theses. These choices were made in spite of the fact that he was a really bright guy and working in a hot research field.

He was apparently a nightmare to work for.

It wouldn't hurt to confirm your first impressions by communicating with people who have worked with and for the advisors you are considering. Former research students, collaborators, et cetera. Being personable and likeable is a moderately positive sign, but there are a lot of different components to being a good PhD mentor.