The solution is to change the cutoff for the adaptive embeddings, as mentioned by user TevenLeScao in this GitHub issue: https://github.com/huggingface/transformers/issues/8098#issuecomment-717914018 20
To summarize, you need to set:

cfg = TransfoXLConfig(cutoffs=[0, x])

Where 0 < x < vocab_size
If you don’t do this, the model will try to generate an embedding size of vocab_size - cutoffs[-1], which for a vocab_size < default vocab size, will be negative and throw an error.