The college was founded as the Milton Academy (high school) by a group of early Milton settlers, including Milton House owner Joseph Goodrich. It eventually grew to encompass 16 buildings spread over 24 acres (97,000 m2). Its music department was renowned, and a high percentage of foreign students for the era kept the student body diverse.