UnicodeDecodeError while loading pretrained model from AutoModel.from_pretrained()
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.0.dev0
- Platform: Ubuntu/Linux
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.0
- Tensorflow version (GPU?): 2.6.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
I wanted to train a custom language model in devangari dataset. I trained the model using google-research  BERT repo using the scripts there.
![lang](https://user-images.githubusercontent.com/15711560/134165890-46627620-2fca-49d7-966d-81a3b9da398d.png)

This was the result after running run_pretraining,py
Then i used the huggingface script convert_bert_original_tf_checkpoint_to_pytorch.py to change the model to pytorch model. 
Pytorch Model was produced  using the script.
Then I used this script to load the model using huggingface TFAutomodel.
from transformers import AutoModel
from transformers import AutoTokenizer
save_directory = "/home/info/Documents/language_model_nlp/bert_language_model/language_model_pytorch/devanagari_language_model.bin"
tokenizer = AutoTokenizer.from_pretrained("/home/info/Documents/language_model_nlp/bert_language_model/devanagari_tokenizer")
model = AutoModel.from_pretrained(save_directory,from_pt=True)
I got error 
![errorr](https://user-images.githubusercontent.com/15711560/134166361-a0ae2774-0680-4bff-a227-d51608cffe0f.png)

The path for the `AutoModel` should be to a directory pointing to a `pytorch_model.bin` and to a `config.json`. Since you're pointing to the `.bin` file directly, the configuration cannot be loaded.<|||||>> The path for the `AutoModel` should be to a directory pointing to a `pytorch_model.bin` and to a `config.json`. Since you're pointing to the `.bin` file directly, the configuration cannot be loaded.

I am giving the path of the model that I got when I ran this script convert_bert_original_tf_checkpoint_to_pytorch.py on the original model produced by training language model. I am confused which model file should I provide on AutoModel.from_pretrained() function.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.