Is HuggingFace TransfoXLLMHeadModels trainable from scratch?
Hello,

Is HuggingFace TransfoXLLMHeadModels trainable from scratch? The documentation makes it look like it is possible to train the TransfoXLLMHeadModel from scratch, since (according to the documentation) the loss can be returned via TransfoXLLMHeadModel( ) as long as labels are provided (https://huggingface.co/transformers/model_doc/transformerxl.html#transfoxllmheadmodel) .

However, the code for TransfoXLLMHeadModels shown in the Github repository 
(https://github.com/huggingface/transformers/blob/master/transformers/modeling_transfo_xl.py#L780) seem to suggest that the loss is, in fact, not returned even when the labels are provided.

Is HuggingFace TransfoXLLMHeadModels trainable from scratch?

Thank you,

Hi @h56cho,

The loss is actually returned if labels are present.

Check https://github.com/huggingface/transformers/blob/master/transformers/modeling_transfo_xl.py#L793<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
