`Wav2Vec2FeatureExtractor` and `Speech2TextFeatureExtractor` tests fail for truncating `max_length` values
# Misleading use of `max_length` in `FeatureExtraction` classes

**TL;DR**

- Currently, setting `max_length` and having features be longer than `max_length` does not necessarily mean it should be `truncation=True`. 
- Further look into testing code makes me believe that this is by design, as explained [below](https://github.com/huggingface/transformers/issues/16419#issuecomment-1079705990).
- I believe this is confusing because padding to `length==X` for shorter sequences and not truncating sequences greater than `X` is what you'd expect if `X` is `min_length`, not `max_length`.


---

The current tests for the existing `FeatureExtractor` classes are using somewhat meaningless `max_length` values that are greater than length of the input features. 

**When using lower values that truncate the input features (for a meaningful effect of `max_length` parameter), the test code fails.**

## Environment info

- `transformers` version: 4.18.0.dev0
- Platform: Linux-3.10.0-1062.el7.x86_64-x86_64-with-debian-10.1
- Python version: 3.7.9
- Huggingface_hub version: 0.2.0
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.1
- JaxLib version: 0.3.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@patrickvonplaten, @anton-l


## Information

Model I am using (Bert, XLNet ...): `Wav2Vec2FeatureExtractor` and `Speech2TextFeatureExtractor`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

### Existing Working Test 1

From `tests/wav2vec2/test_feature_extraction_wav2vec2.py` line 127:
```python
def test_zero_mean_unit_variance_normalization_np(self):
    feat_extract = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]
    max_lengths = [None, 1600, None]
    for max_length, padding in zip(max_lengths, paddings):
        processed = feat_extract(speech_inputs, padding=padding, max_length=max_length, return_tensors="np")
        input_values = processed.input_values

        self._check_zero_mean_unit_variance(input_values[0][:800])
        self.assertTrue(input_values[0][800:].sum() < 1e-6)
        self._check_zero_mean_unit_variance(input_values[1][:1000])
        self.assertTrue(input_values[0][1000:].sum() < 1e-6)
        self._check_zero_mean_unit_variance(input_values[2][:1200])
```


### Failing Test 1 (after changing `max_lengths`)

```python
def test_zero_mean_unit_variance_normalization_np(self):
    feat_extract = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]

    max_lengths = [None, 100, None]

    for max_length, padding in zip(max_lengths, paddings):
        processed = feat_extract(speech_inputs, padding=padding, max_length=max_length, return_tensors="np")
        ....
```

### Existing working test 2

From `tests/speech_to_text/test_feature_extraction_speech_to_text.py` Line 140
```python
def test_cepstral_mean_and_variance_normalization(self):
    feature_extractor = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]
    max_lengths = [None, 16, None]
    for max_length, padding in zip(max_lengths, paddings):
        inputs = feature_extractor(
            speech_inputs, padding=padding, max_length=max_length, return_attention_mask=True
        )
        input_features = inputs.input_features
        attention_mask = inputs.attention_mask
        fbank_feat_lengths = [np.sum(x) for x in attention_mask]

        self._check_zero_mean_unit_variance(input_features[0][: fbank_feat_lengths[0]])
        self._check_zero_mean_unit_variance(input_features[1][: fbank_feat_lengths[1]])
        self._check_zero_mean_unit_variance(input_features[2][: fbank_feat_lengths[2]])
```



### Failing test 2

```python
def test_cepstral_mean_and_variance_normalization(self):
    feature_extractor = self.feature_extraction_class(**self.feat_extract_tester.prepare_feat_extract_dict())
    speech_inputs = [floats_list((1, x))[0] for x in range(800, 1400, 200)]

    paddings = ["longest", "max_length", "do_not_pad"]

    max_lengths = [None, 4, None]

    for max_length, padding in zip(max_lengths, paddings):
        inputs = feature_extractor(
            speech_inputs, padding=padding, max_length=max_length, return_attention_mask=True
        )
        input_features = inputs.input_features
        ....
```
 
For identical error messages:

```
        if self.do_ceptral_normalize:
            attention_mask = (
                np.array(attention_mask, dtype=np.int32)
>               if self._get_padding_strategies(padding, max_length=max_length) is not PaddingStrategy.DO_NOT_PAD
                else None
            )
E           ValueError: setting an array element with a sequence.
```


## Why It Fails

The working tests produce `attention_mask` values like this:
```
[[1 1 1 0 0 0]
 [1 1 1 1 0 0]
 [1 1 1 1 1 1]]
```

While the failing tests produce `attention_mask` values **at the same point in the code** like this:

```
[array([1, 1, 1, 0], dtype=int32) array([1, 1, 1, 1], dtype=int32)
 array([1, 1, 1, 1, 1, 1], dtype=int32)]
```

A very easy fix would just be doing type checking and changing the latter type to the same as the former.

But maybe it's worth looking into `SequenceFeatureExtractor`'s `pad` function that returns `List[np.array]` for truncated outputs and `List[List]]` for untruncated outputs.




<!-- A clear and concise description of what you would expect to happen. -->

So despite the `max_length` parameter, it's not actually truncating (rather, it's forgetting to for the last item):

```
[
array([1, 1, 1, 0], dtype=int32) 
array([1, 1, 1, 1], dtype=int32)
array([1, 1, 1, 1, 1, 1], dtype=int32)
]
```

The above is a list of "ragged" nested sequences, and turns out:

```
feature_extraction_utils.py:172: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a 
list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must 
specify 'dtype=object' when creating the ndarray.
```<|||||>Okay, so it seems like this behavior is by design:
https://github.com/huggingface/transformers/blob/main/tests/test_sequence_feature_extraction_common.py#L244

```python
# truncate to smallest
input_1 = feat_extract.pad(
    processed_features, padding="max_length", max_length=len(speech_inputs[0]), truncation=True
)
input_1 = input_1[input_name]

input_2 = feat_extract.pad(processed_features, padding="max_length", max_length=len(speech_inputs[0]))
input_2 = input_2[input_name]

self.assertTrue(_inputs_have_equal_length(input_1))

# suggests we don't want equal lengths, despite max_length, since we didn't set `truncation=True`
self.assertFalse(_inputs_have_equal_length(input_2))
```

but isn't this confusing?

Rather, the current behavior is akin to `min_length`, where we pad all the inputs that are shorter than `max_length` to be equal to `max_length`, and **not truncate** longer sequences.


Found this while making a small PR to add and adjusting the test codes accordingly:
```python
def pad(self, ....):
    if truncation is False and max_length is not None:
        raise ValueError(
            "If `max_length` is provided, `trunction` must be True."
        )

...

# tests/speech_to_text/test_feature_extraction_speech_to_text.py
paddings = ["longest", "max_length", "do_not_pad"]
max_lengths = [None, 4, None]
for max_length, padding in zip(max_lengths, paddings):
    do_truncation = max_length is not None
    inputs = feature_extractor(
        speech_inputs, padding=padding, max_length=max_length, 
        return_attention_mask=True, truncation=do_truncation
    )
```

Could someone let me know if I should continue?



<|||||>@cwkeam Indeed, this logic might be confusing for sequence feature extraction in particular! 
However, the fact that setting `max_length` doesn't make `truncate=True` stems from the base padding and truncation strategies of `transformers` (see for example [here](https://github.com/huggingface/transformers/blob/b320d87eceb369ea22d5cd73866499851cb2cca3/docs/source/pad_truncation.mdx)). 
Maybe @LysandreJik or @sgugger have a nicer explanation for why that's the case?<|||||>Transformers does not do any magic behind the scenes. If you want truncation, you have to enable it. If you want padding, you have to enable it. `max_length` itself doesn't set any of those because what default are we supposed to take? There are several truncation strategies even when a max length is set (to deal with pairs of sentences for instance).

Our design choice, in accordance with our [philosophy](https://huggingface.co/docs/transformers/philosophy) was to avoid any magic default and let the user manually control everything. I agree it can be confusing and the documentation could definitely be improved (if you wanted to make a contribution ;-) )<|||||>Right, thanks for kindly explaining! I think I had to understand that `padding` and `truncation` must be two well-separated functionalities. That makes sense.

Thank you!