 After shelving his swimming career on his return to Australia for four years, Charlton again broke the Australian record in both the 440 and freestyle events at the 1932 New South Wales championships to gain selection for the 1932 Summer Olympics in Los Angeles, the oldest member of the team at 25 years of age.He contracted influenza a fortnight before the Games after arriving in the United States.Charlton raised hopes that he had recovered when he won his heat, but then only won third place in the semi-final, although he still progressed to the final of the 400 m freestyle.