when not travelling overseas on cricket tours, harvey played baseball in the winter for the fitzroy baseball club. he was twice named in the australian baseball team but the team was named only for the distinction accorded on the players: that is, they never competed. harvey's fielding abilities were regarded by wisden as the "finest outfielder in the world" during his career. as a baseball infielder harvey developed a half round arm throw; its speed and accuracy caused many batsmen to be run out while attempting a run. ray robinson said that harvey's throw was "arrow-like" in accuracy and that "as a versatile fieldsman, this ball-hawk...takes top place". his baseball training also influenced his habit of catching the ball above head height, with which he rarely dropped catches. this was based on the theory that the fielder need never take his eyes off the ball and, if it were to bounce out of his hands, he would have time to attempt to grab the rebound. harvey also covered ground quickly and possessed an efficient method of picking up and returning the ball. from late 1958 when norm o'neill made his test debut until harvey's retirement in 1963, the duo formed a formidable pairing in the covers, helping to restrict opposition batsmen from scoring in the region.