Actually the towers are a host for some of the specialization classes - like Advanced Defense Against the Dark Arts. Also libraries and the duelist areas too. The dorms are mostly housed in the main portion of the castle. Even people with different guardians are mixed.