This restriction reinforces the notion of the imperial family as living descendents and representatives of the gods despite the emperor's renunciation of divinity announced as one of the terms of surrender at the end of World War II (when some people literally fainted upon hearing the emperor's voice on the radio for the first time).
Some people fainted when they heard the emperor's voice.