I don’t think the data loading should be performed in FP16, as you might end up with some quantization noise.
If I’m not mistaken, the 16bit audio files represent 65536 different levels.
Since FP16 cannot represent all integers >2048 (Wikipedia - FP16 1), you’ll lose some information.
That being said, once you’ve loaded and preprocessed the data, you could still use FP16 for the model training. Have a look at apex/amp 2 for an automatic mixed-precision approach.