He was the son of Dr. William Fraser Tolmie, a prominent figure in the Hudson's Bay Company and a member of both the colonial assembly of Colony of Vancouver Island and the Legislative Assembly of British Columbia. William Fraser was early supporter of Scottish industrialist reformer Robert Owen, and was a strong supporter of women's suffrage in British Columbia. His maternal ancestry was Indigenous and representative of the marriages of Indigenous women and French and Scottish men who worked in the fur trade.