Music perceptual abilities are subjective and exhibit high inter-individual variability. Twenty-nine participants with varying degrees of musical training were tested for musical perception ability with the Profile of Music Perception Skills (PROMS) and brain structural measures obtained via diffusion tensor imaging. Controlling for the period of training, TBSS results showed that individuals with better musical perception abilities showed increased deviations from linear anisotropy in the corpus callosum. Specifically, mode of anisotropy in the genu and body of the corpus callosum was negatively correlated with music perception score suggesting the presence of crossing fibers. A multi-compartment model of crossing fibers revealed a significant positive relation for partial volumes of secondary fiber populations with timing aspects of music perception. Our results suggest that inter-hemispheric connectivity differences in the anterior parts of the corpus callosum may reflect innate differences in the processing of the rhythmic aspects of music.