 While at Harvard, Brown earned money as the college organist and unhappily spent some summer months as a village school teacher.Brown befriended and roomed with his Harvard classmate Horatio Alger and counted Ephraim Whitman Gurney (who became a professor of philosophy and history and dean of the Harvard faculty) as his closest college friend.Brown received an Artium Baccalaureus degree in 1852 from Harvard University, ranked second in his class.