Prevent BatchEncoding from blindly passing casts down to the tensors it contains
# What does this PR do?

This PR prevents `BatchEncoding.to` from passing down things which aren't devices to the tensors it contains. Previously it would pass down all the arguments, and as the `to` method in pytorch can also cast the arguments to different types it's used blindly by other packages (e.g. Nvidia's Apex). This caused an issue where when using Apex's AMP support with `O2` or greater it would cast the token indexes from a `LongTensor` to a `HalfTensor` truncating our vocab at 65k and rounding most of the words to the nearest 8th word (if you blindly insert the cast back in in the embedding layer, which the warning says to do).

The doc for `BatchEncoding.to` says it is only for moving the encoding and the tensors it contains between devices, but as the type checking isn't on by default it can behave like a regular pytorch `to` method and accept cast arguments that it passes down to the tensors it contains.

Fixes #6582

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #6582
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

There are no docs or tests changes as the change makes the method conform with its currently documented behaviour.

@LysandreJik


black complained about the style after the update, so I fixed it and squashed the commits again.<|||||>Thank you @Craigacp!