 He was the second son of banker Alexander Brown (1764–1834) and Grace (née Davison) Brown (1759–1843).His elder brother was William Brown (who later became the 1st Baronet of Richmond Hill) and his younger brothers were John Brown and James Brown.He emigrated from Ulster to Baltimore, Maryland, at the age of 15 in 1802.