run_generation.py  'encode' error for gpt2 and xlnet
Hello:

Been using nshepperd's tf repo and various excellent forks for fine-tuning and inference without issue.

Wanted to check out py-torch transformers and compare. First test is simple conditional sampling from the pytorch models:

python3 run_generation.py --model_type=xlnet --length=20 --model_name_or_path=models/xlnet-large-cased

Loads config.json and model weights then Model prompt >>> 

Enter anything: Hello there huggingface. What's up?

then ...

Traceback (most recent call last):
  File "run_generation.py", line 195, in <module>
    main()
  File "run_generation.py", line 175, in main
    context_tokens = tokenizer.encode(raw_text)
AttributeError: 'NoneType' object has no attribute 'encode'

Same error when switching to gpt2, in this case  gpt-large.

Is there a modification to the script required?  Is the terminal syntax incorrect?

Can't move on to fine-tuning until basic inference sorted out.

Tested both pytorch 1.2.0 and 1.1.0.  Same error.

    OS: ubuntu 18.04
    Python version: python 3.6.8
    PyTorch version: torch 1.2.0 tested and changed to 1.1.0 to match transformer
    PyTorch Transformers version (or branch): pytorch-transformers 1.2.0 tested and changed to 1.1.0 to match above
    Using GPU ? yes
    Distributed of parallel setup ? no

All help appreciated. Would like to test distributed fine-tuning but need the basic repo working first on local machine.

Cheers

Closing own non-issue. Needed to download missing files listed in the four applicable tokenizer scripts. Working 100%. On to fine-tuning.