UnicodeDecodeError:
Traceback (most recent call last):
  File "run_classifier_br.py", line 1061, in <module>
    main()
  File "run_classifier_br.py", line 772, in main
    tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 197, in from_pretrained
    tokenizer = cls(resolved_vocab_file, *inputs, **kwargs)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 97, in __init__
    self.vocab = load_vocab(vocab_file)
  File "/home/luwei/anaconda3/lib/python3.6/site-packages/pytorch_pretrained_bert/tokenization.py", line 56, in load_vocab
    token = reader.readline()
  File "/home/luwei/anaconda3/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

@ZhaoxinRuc 

I'm assuming that some data in your `vocab.txt` file contains bad characters which your `codecs.py` can't decode properly. When I downloaded the pre-trained weights folder, it came with `vocab.txt` which didn't had this issue. Check if you've downloaded a wrong version or some how changed contents of `vocab.txt`.

It's also likely that your operating system actually messed with the encoding of certain characters in your `vocab.txt` file. For that, try saving the text file in 'utf8' format , just to make sure.

Hope it helps.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
