Question: Computer Science PhD admit here. 

Is it crazy of me to turn down a top school (MIT/Stanford) for a lower-ranked school (think Cornell/UW/Columbia/Michigan) if I think there is better advisor fit?

Obviously all of these are fantastic schools and I am very lucky to be choosing from them. However, I felt like I got along really well with my would-be advisor at the lower-ranked school than I did at MIT/Stanford. People have told me that it's crazy of me to turn down MIT/Stanford since they are, along with Berkeley/CMU, on a different "level" than the other schools.

(My field is somewhat narrow so there is only 1~2 faculty at each of the schools doing research in the area. So regardless of which school I go to, if the advisor doesn't work out, I would be in a dire position.)

EDIT: Both advisors are well known in the field (and both have produced outstanding graduates), but the MIT/Stanford professor is good deal more senior. Also, talking with their current students, the MIT/Stanford professor is known to be pushy and have a "strong" personality (which may work for some people, but not sure whether it will work for me, as I've never worked with such people before).

Answer: 
There are some aspects of the decision that are independent of your advisor, such as having a large pool of impressive peers to learn from and work with, but the degree of fit with your advisor is certainly a major factor.

If you believe you can't work effectively with the potential advisors at the higher-ranked schools, then there's no point in choosing one of those schools. If the advisor at the lower-ranked school is performing at or above the level of the other possible advisors (as judged by influential papers and successful students), then again you don't have to worry. But your phrasing makes me a little uncomfortable:

> 
> I felt like I got along really well with my would-be advisor at the lower-ranked school
> 
> 
> 

This sounds like you are describing how quickly you clicked and became comfortable, but you can learn an enormous amount from someone even if you never develop an easy rapport with them or feel you could be friends. It's worth thinking about whether the less comfortable relationship could nevertheless be productive. Maybe it can't - it depends on the personalities and the details of your interaction - but it's important to distinguish how much you like the advisor from how fruitful the intellectual relationship might be.

I wouldn't stress out too much about this decision, though. My impression is that Cornell and UW are not so markedly far behind MIT and Stanford in computer science. (There's a gap, but not a huge gap.) If one of them feels like a better fit, you are not jeopardizing your career.
All of the above answers are correct. However, do trust your first impressions. I didn't click with my first prospective advisor and picked someone else. That prospective advisor failed to get tenure after his next couple of graduate students opted to switch advisors and start new research projects in the 3-4th year of their theses. These choices were made in spite of the fact that he was a really bright guy and working in a hot research field.

He was apparently a nightmare to work for.

It wouldn't hurt to confirm your first impressions by communicating with people who have worked with and for the advisors you are considering. Former research students, collaborators, et cetera. Being personable and likeable is a moderately positive sign, but there are a lot of different components to being a good PhD mentor.