Chan was born on 6 February 1953, in Hong Kong with his brother Chan Lung. At the age of 7, he attended the Nah-Hwa Primary School on Hong Kong Island, where he failed his first year, after which his parents withdrew him from the school.