 Ballard was born and raised in the Shanghai International Settlement, an area under foreign control where people "lived an American style of life".He was sent to the Cathedral School, the Anglican Holy Trinity Church near the Bund, Shanghai.After the outbreak of the Second Sino-Japanese War, Ballard's family were forced to evacuate their suburban home temporarily and rent a house in central Shanghai to avoid the shells fired by Chinese and Japanese forces.