Using HuggingFace TransfoXLLMHeadModel() with custom Torchtext vocabulary
Hello,
I am trying to use the HuggingFace TransfoXLLMHeadModel on WikiText2 dataset under a customized TransfoXLConfig with different vocabulary, and it causing an error. I am not sure how to fix it. Below are my code:

```js

# Import packages 
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import TransfoXLConfig, TransfoXLTokenizer, TransfoXLModel, TransfoXLLMHeadModel, TFTransfoXLModel, TFTransfoXLLMHeadModel
import spacy
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.data import Field, BPTTIterator, TabularDataset 
import math
import random
import numpy as np
import pandas as pd 
import time

# define the English text field
TEXT = Field(tokenize = 'spacy',
             init_token='<sos>',
             eos_token='<eos>',
             tokenizer_language='en',
             lower=True)

# load WikiText-2 dataset and split it into train and test set
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)

# build vocabulary based on the field that we just defined.
TEXT.build_vocab(train_Wiki2, val_Wiki2, test_Wiki2)

# get number of tokens
ntokens = len(TEXT.vocab.stoi) # ntokens =  28871

# define transformer-XL configuration.
transfoXLconfig = TransfoXLConfig(vocab_size_or_config_json_file = ntokens,
                                  cutoffs = [20000, 40000, 200000], 
                                  d_model = 64, 
                                  d_embed = 64, 
                                  n_head = 16, 
                                  d_head = 64,
                                  n_layer = 5,
                                  attn_type = 0,
                                  dropout = 0.1, 
                                  output_hidden_states = True,
                                  output_attentions = True)

# define the transformer-XL model based on the specified configuration.
model = TransfoXLLMHeadModel(transfoXLconfig) # this line is causing an error.
"""
Error message: 
Traceback (most recent call last):

  File "<ipython-input-14-fa91df67f439>", line 1, in <module>
    model = TransfoXLLMHeadModel(transfoXLconfig)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 818, in __init__
    self.transformer = TransfoXLModel(config)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 599, in __init__
    div_val=config.div_val)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 421, in __init__
    self.emb_layers.append(nn.Embedding(r_idx-l_idx, d_emb_i))

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 97, in __init__
    self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))

RuntimeError: Trying to create tensor with negative dimension -171129: [-171129, 1]
model = TransfoXLLMHeadModel(transfoXLconfig)
"""
```
How can I use HuggingFace TransfoXLLMHeadModel( ) with a custom vocabulary of different size?

Thank you,


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>I faced the same problem:I solved it by passing the size of your vocabulary (from your custom tokenizer) as a parameter.
I proceeded as follows:
`vocabulary_size = tokenizer.vocab_size`
```
configuration = tf.TransfoXLConfig(vocab_size_or_config_json_file=vocabulary_size, cutoffs=cutoffs,
                                       d_model=512, d_embed=512, n_head=8, d_head=64, n_layer=12, d_inner=2048)
```
I hope that helped :)<|||||>P.s. What do you pass as inputs and labels?
For now, I create a batch as follows:
"The quick brown fox jumps over the lazy dog"
If I have batch_size=2, and sequence length=4:
["The quick brown fox",
"jumps over the lazy"]

What do you feed to the Transformer-XL as input?