Problem loading finetuned XLNet model
After fine-tuning an XLNet classification model and obtaining TF checkpoints I converted the checkpoint to pytorch_model.bin and config.json. I need to make prediction on input text, but I have problems loading the models correctly. Any help?
What task did you fine-tuned it on?
You can convert it by running the `convert_xlnet_checkpoint_to_pytorch.py` script with a `--finetuning_task` argument (see [here](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/convert_xlnet_checkpoint_to_pytorch.py#L94-L97))
<|||||>binary classification (sentiment) on my dataset
I tried the following
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetModel
config = XLNetConfig.from_pretrained('./')
tokenizer = XLNetTokenizer.from_pretrained('./')
model = XLNetModel(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]`
however, the output is not binary, the output tensor is :
tensor([[[-1.0645, -1.1170, -0.5549,  ...,  0.7089,  0.2862, -0.2995],
         [ 0.0198, -1.6598, -0.1760,  ...,  0.1167, -0.1545,  0.1777],
         [ 0.6514, -0.5614, -1.2180,  ...,  1.0796,  1.1217,  1.0362]]],
       grad_fn=<PermuteBackward>)

<|||||>the converting was done using:
`pytorch_transformers xlnet $TRANSFO_XL_CHECKPOINT_PATH $TRANSFO_XL_CONFIG_PAH $PYTORCH_DUMP_OUTPUT . `

I did not specified the task, because I though that it is not important.<|||||>It is important to match the last (classification) layer otherwise the conversion will fail.
If you give me more information on your TF training (like which script you usde for instance) I may be able to help.<|||||>I was using the:
`train_command = "python xlnet/run_classifier.py \
  --do_train=True \
  --do_eval=True \
  --eval_all_ckpt=True \
  --task_name=spam \
  --data_dir="+DATA_DIR+" \
  --output_dir="+OUTPUT_DIR+" \
  --model_dir="+CHECKPOINT_DIR+" \
  --uncased=False \
  --spiece_model_file="+PRETRAINED_MODEL_DIR+"/spiece.model \
  --model_config_path="+PRETRAINED_MODEL_DIR+"/xlnet_config.json \
  --init_checkpoint="+PRETRAINED_MODEL_DIR+"/xlnet_model.ckpt \
  --max_seq_length=128 \
  --train_batch_size=8 \
  --eval_batch_size=8 \
  --num_hosts=1 \
  --num_core_per_host=1 \
  --learning_rate=2e-5 \
  --train_steps=4000 \
  --warmup_steps=500 \
  --save_steps=500 \
  --iterations=500"

! {train_command}`
It is similar to the imdb sentiment task
The colab is here
https://colab.research.google.com/drive/1nfWEEDxPOE8myb-hwGdoXs9nVXM3AVcz<|||||>I have modified the:
class ImdbProcessor(DataProcessor):<|||||>Here is the model after model.eval():
`XLNetModel(
  (word_embedding): Embedding(32000, 1024)
  (layer): ModuleList(
    (0): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (1): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (2): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (3): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (4): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (5): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (6): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (7): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (8): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (9): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (10): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (11): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (12): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (13): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (14): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (15): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (16): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (17): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (18): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (19): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (20): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (21): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (22): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
    (23): XLNetLayer(
      (rel_attn): XLNetRelativeAttention(
        (layer_norm): XLNetLayerNorm()
        (dropout): Dropout(p=0.1)
      )
      (ff): XLNetFeedForward(
        (layer_norm): XLNetLayerNorm()
        (layer_1): Linear(in_features=1024, out_features=4096, bias=True)
        (layer_2): Linear(in_features=4096, out_features=1024, bias=True)
        (dropout): Dropout(p=0.1)
      )
      (dropout): Dropout(p=0.1)
    )
  )
  (dropout): Dropout(p=0.1)
)
`<|||||>You'll have to modify the conversion script because this is not a standard task and the number of labels won't be found in the list (see top of the conversion script) you should add `'spam': 2` in the list (if you have two labels indeed).

The other option is to directly load the TF model in PyTorch and save the pytorch model afterwards with something like this:
```
config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='spam')
model = XLNetForSequenceClassification.from_pretrained('path/to/your/tf/model.ckpt.index', config=config, from_tf=True)
model.save_pretrained('pytorch_model_saving_directory')
```<|||||>Perfect, tnx.
Just a minor question if I have classification model with 4 labels, I do the same changes spam:4?<|||||>Yes, change my `2` to `4`<|||||>Hi Thom,
I did:
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetForSequenceClassification
config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='sentiment')
model = XLNetForSequenceClassification.from_pretrained('../model.ckpt-2500', config=config, from_tf=True)
model.save_pretrained('./test')`

Afterwards I get in the test folder the config.json and pytorch_model.bin.

I tried to run:
'import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetModel
config = XLNetConfig.from_pretrained('./')
tokenizer = XLNetTokenizer.from_pretrained('./')
model = XLNetModel(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]
print(last_hidden_states)


There was an error:
Model name './' was not found in model name list (xlnet-base-cased, xlnet-large-cased). We assumed './' was a path or url but couldn't find tokenizer filesat this path or url.

So I included the spiece.model into the same test folder, but still the results is:
tensor([[[ 0.0156, -0.9046,  0.9789,  ..., -0.7764,  1.1309,  0.2862],
         [ 0.4416,  0.0665,  2.0020,  ...,  0.4117,  0.9779, -1.0588],
         [ 0.6908, -0.0000,  0.1479,  ...,  0.0032,  0.9871, -0.9482]]],
       grad_fn=<PermuteBackward>)

Do u know what I am doing wrong? <|||||>While converting this is some of the warnings:
Weights not copied to PyTorch model: beta1_power, beta2_power, global_step, model/classification_finsent/logit/bias, model/classification_finsent/logit/bias/Adam, model/classification_finsent/logit/bias/Adam_1, model/classification_finsent/logit/kernel, model/classification_finsent/logit/kernel/Adam, model/classification_finsent/logit/kernel/Adam_1<|||||>In addition, I have tried the other approach, to modify the script, but the results are the same. I think that the problem is that the weights from the classification_finsent are not copied to pytorch model. Any suggestions on this<|||||>I have tried also in this way:
`import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification

config = XLNetConfig.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/PyTorch/pytorch_transformer_script/test')
tokenizer = XLNetTokenizer.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/PyTorch/pytorch_transformer_script/test')
config.output_hidden_states=True
model = XLNetForSequenceClassification(config)

input_ids = torch.tensor(tokenizer.encode("Apple stocks increase rapidly")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
print(outputs[0])`

and the output now is:
tensor(0.3944, grad_fn=<NllLossBackward>)
<|||||>In the last attempt I did:
```python
import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification

config = XLNetConfig.from_pretrained('xlnet-large-cased', num_labels=2, finetuning_task='finsent')
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased') #I am using the default tokenizer

model = XLNetForSequenceClassification.from_pretrained('/home/igor/Falcon/XLNet/sentiment_model/model.ckpt-2500',config=config, from_tf=True) #this is my finetuned model
model.eval() #in evaluation mode
input_ids = torch.tensor(tokenizer.encode("Apple stock increase and they are overwhelmed with the success")).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, logits = outputs[:2]
print(torch.nn.functional.softmax(logits.data))
```

However I am getting different prediction for the same input data, I did model.eval() to stop the dropout, but still the inference looks random.

In addition here is my config file printed:
```json
{
  "attn_type": "bi",
  "bi_data": false,
  "clamp_len": -1,
  "d_head": 64,
  "d_inner": 4096,
  "d_model": 1024,
  "dropatt": 0.1,
  "dropout": 0.1,
  "end_n_top": 5,
  "ff_activation": "gelu",
  "finetuning_task": "finsent",
  "init": "normal",
  "init_range": 0.1,
  "init_std": 0.02,
  "initializer_range": 0.02,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "mem_len": null,
  "n_head": 16,
  "n_layer": 24,
  "n_token": 32000,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "reuse_len": null,
  "same_length": false,
  "start_n_top": 5,
  "summary_activation": "tanh",
  "summary_last_dropout": 0.1,
  "summary_type": "last",
  "summary_use_proj": true,
  "torchscript": false,
  "untie_r": true
}
```
It looks that I getting closer, but still the inference is strange<|||||>@thomwolf I finally succeed to import the checkpoint model and infer. Still I am not sure if this a valid approach:
```python
import torch
from pytorch_transformers import XLNetConfig
from pytorch_transformers import XLNetTokenizer
from pytorch_transformers import XLNetForSequenceClassification


seed = 0
torch.manual_seed(seed)
if torch.cuda.is_available():
	torch.cuda.manual_seed_all(seed)
#load the initial config file from the XLNet model
config = XLNetConfig.from_pretrained('xlnet_config.json', num_labels=2, finetuning_task='finsent')
#the tokenizer I am using is the initial one (spiece.model)
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')

model = XLNetForSequenceClassification.from_pretrained('/sentiment_over_imdb/model.ckpt-4000',config=config, from_tf=True)
model.eval()

def sentiment(data):
	input_ids = torch.tensor(tokenizer.encode(data)).unsqueeze(0)  # Batch size 1
	labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1

	outputs = model(input_ids, labels=labels)
	loss, logits = outputs[:2]
	output = torch.nn.functional.softmax(logits.data, dim=1)
	sent = output.tolist()[0][1]
        return sent
```