 Smith was born in Dalkeith Parish, Midlothian, Scotland, into a family that included many masons.As a young man he apprenticed in the building trades, and emigrated to America, either directly from Scotland or via London it is unknown, in late 1748.Smith quickly became a member of the Carpenters' Company of Philadelphia, and is considered by many to be the foremost master-builder, or carpenter-architect, of the Colonial Period.