Make OpenAIGPTTokenizer work with SpaCy 2.x and 3.x
SpaCy 3.x introduced an API change to creating the tokenizer that
breaks OpenAIGPTTokenizer. The old API for creating the tokenizer in
SpaCy 2.x no longer works under SpaCy 3.x, but the new API for creating
the tokenizer in SpaCy 3.x DOES work under SpaCy 2.x. Switching to the
new API should allow OpenAIGPTTokenizer to work under both SpaCy 2.x and
SpaCy 3.x versions.

Fixes https://github.com/huggingface/transformers/issues/14449


I'm not able to test the changes locally. When I run `pip install -e ".[dev]"`, I encounter a build error and the install fails. It doesn't look like pytest coverage is sufficient here either since SpaCy and ftfy are not required by transformers, and the code can execute correctly without them being installed.<|||||>@patrickvonplaten @LysandreJik Not sure why but I can't tag you as reviewers on GitHub, so tagging you in comments here.<|||||>Yes, why not! GPT-2 is among our most used models, so I think testing that the tokenization behaves correctly across all possibilities is important. Would you like to take a stab at it @cody-moveworks?

In order to do so, you could start by adding an `is_spacy_available` method in `file_utils.py`, analog to other methods such as `is_vision_available` here: https://github.com/huggingface/transformers/blob/2e9af294940083915ccb2740a7c8d5b154194f15/src/transformers/file_utils.py#L507-L508

Then it would require defining a `require_spacy` unittest decorator in `testing_utils.py`, such as the `require_vision` here: https://github.com/huggingface/transformers/blob/2e9af294940083915ccb2740a7c8d5b154194f15/src/transformers/testing_utils.py#L404-L412

Thirdly, you can add a test in the `tests/test_tokenization_gpt2.py` file with the `@require_spacy` decorator, which will only run when SpaCy is installed.

And finally, we can modify this CircleCI run: https://github.com/huggingface/transformers/blob/2e9af294940083915ccb2740a7c8d5b154194f15/.circleci/config.yml#L538-L565

So that it:
1. Installs SpaCy
2. Runs the tokenization GPT-2 test file!<|||||>@LysandreJik Thanks for the detailed walkthrough of the changes needed to add testing. I'll take a stab at it.<|||||>@LysandreJik I made the changes and all checks are passing. Can you take a look?