Bert model instantiated from BertForMaskedLM.from_pretrained('bert-base-uncased') and BertForMaskedLM(BertConfig.from_pretrained('bert-base-uncased')) give different results 
The two different methods for instantiating a model produce different losses.

`from pytorch_transformers import BertForMaskedLM, BertConfig, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  

config = BertConfig.from_pretrained('bert-base-uncased')
config_model = BertForMaskedLM(config)
config_model.eval()
with torch.no_grad():
    config_outputs = config_model(input_ids, masked_lm_labels=input_ids)
config_loss = config_outputs[0]
print(config_loss.item())

pretrained_model = BertForMaskedLM.from_pretrained('bert-base-uncased')
pretrained_model.eval()
with torch.no_grad():
    pretrained_outputs = pretrained_model(input_ids, masked_lm_labels=input_ids)
pretrained_loss = pretrained_outputs[0]
print(pretrained_loss.item())
assert config_loss.item() == pretrained_loss.item()`

The losses produced:
10.574708938598633
1.690806269645691

Hi, not only they give different results, but also BertModel(BertConfig.from_pretrained('bert-base-uncased')) will give a different result each time you run it. **Other bert models also have this problem**; I think this is a bug. @thomwolf 

Following code works well and produce the same result each time you run it.
___________________________
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertConfig
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
a = np.squeeze(outputs[0].detach().numpy())
avg = np.mean(a,axis = 0)
print(avg[0])


**Above code will always output -0.2769656.**
___________________________
Following code will produce a different result each time:
__________________________
config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel(config)
model.eval()
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) 
outputs = model(input_ids)
last_hidden_states = outputs[0]
a = np.squeeze(last_hidden_states[0].detach().numpy())
avg = np.mean(a,axis = 0)
print(avg[0])<|||||>It seems that BertModel(config) returns a random intialized Bert model with the architecture as the config file indicates, because the __init__ function doesn't load pretrained weights. BertModel.from_pretrained() is the right function to load both model architecture and pretrained weights. It's the same for other bert classes.<|||||>Shouldn't BertConfig.from_pretrained('bert-base-uncased') return a config that loads pretrained weights instead of randomly initialized ones?  I thought was the whole point of the example code in the [docs](https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertformaskedlm):


config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM(config)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]<|||||>It should return a config that loads pretrained models; however it does not act like this way.


Sent from Yahoo Mail for iPhone


On Sunday, August 4, 2019, 4:32 PM, Christian Storm <notifications@github.com> wrote:


Shouldn't BertConfig.from_pretrained('bert-base-uncased') return a config that loads pretrained weights instead of randomly initialized ones? I thought was the whole point of the example code in the docs:

config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM(config)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.



<|||||>You are right, the example in the doc is misleading.

The only way to load pretrained weights in a `model` is to call a `model_class.from_pretrained()` method. I'll fix the doc.<|||||>I've fixed the examples of loading pretrained models in docstrings :-) #973 <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
