It's not because our ears are trained for it, it's because the correlation between two white noise signals have a higher peak at it's delay, making it easier to tell direction. It has nothing to do with training our ears, it's simply correlation factor.