	Patterson's father was an owner of Standardbred horses and as young man, Charles was a harness racing driver. In 1891 his interests shifted to Thoroughbred racing and he was hired to train the horses of renowned Kentucky owner and breeder John Madden.