harvey was the fifth of six boys born to horace harvey. despite his small build, harvey was born large, weighing in at . the family lived in broken hill, where horace was a miner, before moving to sydney, and finally to melbourne in 1926, where they settled in the inner northern industrial suburb of fitzroy. there the six boys were taught cricket under the guidance of their father. in conditions conducive to producing batsmen rather than bowlers, they played cricket using a tennis ball on cobblestones or a marble rebounding from the backyard pavement. the boys went to george street state school and falconer street central school. cricket and cricket talk was an integral part of the daily family life. horace held the family batting record with 198 for broken hill, and continued to play in melbourne club cricket. harvey's eldest brother merv went on to play one test for australia, while mick and ray both played for victoria. all six brothers, the other two being brian and harold also played for fitzroy in district cricket. except for harold, all five represented victoria in baseball.