[SequenceFeatureExtraction] Move padding logic from pure python to numpy
Currently the padding for sequential feature extraction: https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/feature_extraction_sequence_utils.py#L38 is done in pure Python even though the two feature extractors that rely on sequential both always return either a single numpy (or PT or TF) tensor or a list of numpy tensors - see:

- https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L205 for Wav2Vec2
- https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/models/speech_to_text/feature_extraction_speech_to_text.py#L209 

More specifically: 

1) `feature_extractor(raw_audio, return_tensors=None, padding=True)` => returns a list of  numpy arrays
2) `feature_extractor(raw_audio, return_tensors="np", padding=True)` => returns a 2D numpy array (PT and TF return a 2D array in PT and TF correspondingly)

For some background the text tokenizers all have the option to return "pure" Python lists which was then implemented analogous for the speech feature extractors. However it was decided that the each individual sequence will always have be a numpy array. This is because a very common use case is to load audio files via `torchaudio` or `librosa` which return a list of numpy arrays and then run the feature extractor on it. Running `feature_extractor(list_of_numpy_arrays, padding=True)` should then not change the inner numpy arrays to pure python as this would be confusing for the user and also very be very slow. 

Therefore I think the current design is good as is and that we can move the padding logic in https://github.com/huggingface/transformers/blob/9d60eebeb52ed3c266ab8e0cc6871ebeb08a5bc1/src/transformers/feature_extraction_sequence_utils.py#L38 from "pure" Python to a numpified approach.. This would both improve speed and prevent instability issues like: https://github.com/huggingface/transformers/pull/13538 . This won't change anything in the user-facing API since at the moment the the "inner" output is always a numpy vector anyways. 

Do you agree on this @LysandreJik @sgugger @anton-l @patil-suraj  ?

@anton-l - are you maybe interested in giving this PR a try? 
I agree with this proposition.<|||||>Yeah, I'll take this one, thanks for the pointers @patrickvonplaten :+1: <|||||>Very much agree! +1<|||||>This sounds great!