Remove docs only check
Remove the docs only check as it can result in [crashes](https://app.circleci.com/pipelines/github/huggingface/transformers/17209/workflows/d3807ea6-9697-4699-a114-98e6b4d2c4d0/jobs/135576).

Will revert if you disagree @stas00. 