ayres was married three times. he was married to actress lola lane from 1931 until 1933, and to actress ginger rogers from 1934 until 1940, whom he met while starring in the film "don't bet on love" (1933).