The main goal would be to have your data in a format that supports lazy loading. I don’t know enough about audio files to give a great answer on what the solution would be.
If you were to use the h5 format, I would do something along the lines of creating a new member of the root group per file: /file1, /file2, ... or allocate enough room in the file and have each file write to it’s designated portion. This might take a while, but it should be doable (and it would be a one time thing).
I do want to say that HDF5 may not be the answer, you’re going to want to look for any method that allows you to load a subset of the wav file without loading everything. Something like that should be ideal for your task.