Many thanks @Sayak_Paul ! The representative dataset seems to be the issue!
I just use this method:
def representative_data_gen():
     for input_value in tf.data.Dataset.from_tensor_slices(test_audio).batch(1).take(250):
          yield [input_value]

When I change the values e.g. from 100 to 250 the predictions change drastically (for the worse). Is there any method or formula to figure out how large the representative dataset should be? I also tried quantization-aware training and could not get a better result. Therefore I thought it cannot be something in the actual quantization process. But I was wrong with that assumption.