memory crash with large dataset
Hello,

I am using the basic `sentiment-classification` pipeline based on https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english/tree/main.

I was able to predict the sentiment of several hundreds of sentences but ran into trouble when I tried to predict the sentiment of about 9M short sentences stored in a Pandas dataframe. I have a `RTX 3090` and `150GB` of RAM so I think the prediction should work.

Specifically, I tried the create the sentiment labels by running

```
classifier = pipeline(task = 'sentiment-analysis')
df['mylabels'] = [o['label'] for o in classifier(df.text.tolist())]
```
(where `df.text` contains my headline) hoping to take advantage of the batch processing in `classifier` but after a while (1 hour or so) Python crashed after mentioning

```
successfully opened dynamic library cublas64_10.dll
memory allocation of 11976 bytes failed
```

Is this a bug? Is this the correct way to process large dataframes?
Thanks!



Thanks!
on the task manager I see that the dedicated GPU memory usage is constant at 24GB while the shared GPU memory usage is at zero. CPU is at 20% and RAM fills entirely up to 160GB. I cannot share the data (proprietary) but maybe there is something obvious that I am missing here in terms of `pipeline` and processing tricks?

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.