Possibility of disabling add_pooling_layer that works for all models
# 🚀 Feature request

A way to ensure that no additional pooling layers are added that works for all model types.

## Motivation

I just want a barebone transformer and I need the per-token representation. Hence the pooling layer that is added in BERT-based models is not necessary for me, but it still occupies memory and forward pass time. Now, I think this can be disabled via `AutoModel.from_pretrained(name, add_pooling_layer=False)` (right?), but I don't think this would work for all models since some don't have this flag (e.g., GPT-2).
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Commenting for activity<|||||>The GPT-2 doesn't have a pooling layer, so what would you like to disable?<|||||>The issue is that if I pass `add_pooling_layer=False` to GPT-2, it will break. But I want a model-agnostic way to create a base model and ensure that there is no pooling layer added.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Commenting for activity<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.