After shelving his swimming career on his return to Australia for four years, Charlton again broke the Australian record in both the 440 and 880 yards freestyle events at the 1932 New South Wales championships to gain selection for the 1932 Summer Olympics in Los Angeles, the oldest member of the team at 25 years of age.