Output of BertModel does not match fixed feature vectors extracted from the last hidden layer  
I want to finetune bert in my task. However, the bert ouput does not match the fixed feature vectors.

I extracted the fixed feature vectors like this:
I use `extract_features.py` to  extract the fixed feature vectors of last hidden layer (layer -1). The command line is below:
`python extract_features.py --input_file=input.txt --output_file=output.json --vocab_file=model_path/vocab.txt --bert_config_file=model_path/bert_config.json --init_checkpoint=model_path/bert_model.ckpt --layers=-1 --max_seq_length=128 --batch_size=8`


The bert model output extracted like below:
The load model code is like below:
 `model_dict = model.state_dict()`
`#load pretrained released bert model`
  `pretrained_dict = torch.load( 'pytorch_model.bin')`
  `pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}`
`#update params`
  `model_dict.update(pretrained_dict)`
  `model.load_state_dict(model_dict)`

The init model code is like below:
`class A(nn.Module):`
` def __init__(self):`
` self.config = BertConfig.from_json_file('config.json')`
` self.bert = BertModel(self.config)`
`def inference(self, input_ids):`
` all_encoder_layers, _ = self.bert(input_ids, token_type_ids=None, attention_mask=input_mask)`
`return all_encoder_layers[-1]`
(I've omitted some irrelevant code.)

Then I ouput tensor all_encoder_layers[-1]. 
all_encoder_layers[-1] doesn't match the feature vectors extracted by `extract_features.py` .

I checked the params in my model. Bert params have been loaded and it is consistent with pretrained params. Also. the input sequence is consistent.
 Can anybody help me? Is there any settings I forget?





Hi, have you solved this problem?
And does anyone know the order of all_encoder_layers?
Thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
