Understanding feature creation 
## ❓ Questions & Help

Hi, 

I am trying to understand feature creation when using run_lm_finetuning.py. If I let the script download the tokenizer files, everything works fine. If I manually assign a folder for the tokenizer with vocab.json and merges.txt downloaded from Hugging Face,  I get an empty dataframe during feature creation. 

`11/19/2019 16:36:20 - INFO - __main__ -   Creating features from dataset file at /data/test
11/19/2019 16:41:53 - INFO - __main__ -   Saving features into cached file /data/test/roberta-base_cached_lm_999999999998_moleculenet_roberta_train.csv
Traceback (most recent call last):
  File "run_lm_finetuning.py", line 558, in <module>
    main()
  File "run_lm_finetuning.py", line 510, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 175, in train
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
  File "/usr/local/conda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 94, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0`

Just to clarify, I am asking this because I originally tried to create a custom vocab/merges files by giving a new path for tokenizer.from_pretrained for my new files. Failure to do so led me to notice that giving a new path for the original files downloaded from S3 also fail in the new path. If I put my custom files to the path used for the S3 cache (with identical filenames used by the cache, everything works fine.  In other words, if I replace the contents of the following files with my custom content, everything works:

`11/19/2019 18:31:15 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json from cache at /root/.cache/torch/transformers/d0c5776499adc1ded22493fae699da0971c1ee4c2587111707a4d177d20257a2.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b
11/19/2019 18:31:15 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-merges.txt from cache at /root/.cache/torch/transformers/b35e7cd126cd4229a746b5d5c29a749e8e84438b14bcdb575950584fe33207e8.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda`  <|||||>Have you learned anything about this? I'm currently running to the same issue trying to specify an existing model. I noticed that the cache_lm_* file that it creates is only 5 bytes, and gedit says the files is corrupt.<|||||>Yes, so bizarrely even if you have the original hugggingface files in a new location, an empty data frame (the cache_lm file) will be created. Try replacing the original hugging face files downloaded from S3 with your custom vocab/merges files. They should be in /root/.cache/torch/transformers

If you still run into the same problem, then either your data is too small or the vocab/merges files are not in the correct format.<|||||>@AVSuni I get the same issue when using a checkpoint generated by huggingface/transformers. Even when using cached versions of the merge/vocab files from `/root/.cache/torch/transformers/`. Even when I delete the `cache_lm_*` file, specifying the model rather than giving a model name to be used causes this issue.<|||||>Maybe I'm being superstitious, but setting a block size seems to alleviate the issue. I didn't bother tracing the code, but I did notice that when I set `block_size`, the cache file is set with the number of the block size. When I do not set the block size, the temp file has some absurdly long number in the cache file where the block size would be. I don't really see that documented anywhere, but here we are.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
