Trying to Use AlbertTokenizer With my own custom Vocab file
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
## Details
<!-- Description of your issue -->
Good Afternoon All,

I am trying to used AlbertTokenizer to tokenize my corpus with a custom vocab file.  I tried the command :  **AlbertTokenizer("my-custom-vocab", do_lower_case=True, keep_accents=False, bos_token='[CLS]', eos_token='[SEP]', unk_token='<unk>', sep_token='[SEP]', pad_token='<pad>', cls_token='[CLS]', mask_token='[MASK]', )**  but it gives an error : " **_RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())]"_**  . a subser of my Vocab file is attached, what am I doing wrong?

Regards, 
[vocab.txt](https://github.com/huggingface/transformers/files/4246094/vocab.txt)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
Hi, our Albert implementation only handles SentencePiece vocabulary.<|||||>Thank you for your response.  I tried generating a sentencepiece vocab model via ..**_spm.SentencePieceTrainer.Train('--input=ALBERT_PEP3V.txt
--model_prefix=albertv1 --vocab_size=10000 --hard_vocab_limit=False')_**....and a model is successfully generated, however when I run your tokenization all sentences get the exact same encoding; for example **_" I am running" will be [ 0, 3, 0, 3]_** and .....**_" the dog jumps" will be the same [0,3, 0, 3]_**. Any idea why this is happening?

Thanks again.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Can we use higgingface for sentence pair classification?



Ché Martin Ph.D
T: 917-684-0864
E: cmartin@gradcenter.cuny.edu
LinkedIn: https://www.linkedin.com/in/chemartin
________________________________
From: stale[bot] <notifications@github.com>
Sent: Wednesday, April 29, 2020 10:09:51 AM
To: huggingface/transformers <transformers@noreply.github.com>
Cc: Che Martin <cmartin@gradcenter.cuny.edu>; Author <author@noreply.github.com>
Subject: Re: [huggingface/transformers] Trying to Use AlbertTokenizer With my own custom Vocab file (#2996)


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/2996#issuecomment-621235452>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOUU5C3VU3LKUY5A4X5URUTRPAYK7ANCNFSM4K2ODLBA>.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
