Hidden states of BertForPreTraining (load from tf ckpt of google original bert) not exactly equal to the output of extract_features.py in google original bert
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: ubuntu 20.04
- Python version: 3.8
- PyTorch version (GPU?): 4.10.2
- Tensorflow version (GPU?): 2.4.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik 

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.



Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load bert-base-chinese official ckpt to pytorch BertForPreTraining
2. get hidden states of -2 (or -1) layer
3. run extract_features.py to get -2 (or -1) layer hidden states of original ckpt model with tensorflow
4. the values of two sides are not  exactly the same with some bias

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
a pytorch bert load from google ckpt can get the same outputs with the original tensorflow bert
<!-- A clear and concise description of what you would expect to happen. -->

Hi,

Would you mind elaborating an approximate range of the bias? Also, is the hidden states output of BertForPreTraining deterministic?

If the output is deterministic but slightly different than TensorFlow's output (the differences are smaller than roughly 1e-5), this is probably a normal behavior due to different BLAS implementations dependent on platform, framework...etc.

See also: https://github.com/pytorch/pytorch/issues/9146#issuecomment-409331986<|||||>@qqaatw  thank you for answering. Yes, the hidden states output of both BerForPreTraining and google-research bert are deterministic.
Sometimes the bias is small, may be 1e-3~1e-4, while using chinese_L-12_H-768_A-12, as showing below:
google-research:   [0.41173,  0.086385,  0.705549, 0.224586, 0.751009, -1.071174, -0.455632, -0.390582, -0.523216, 0.520333,...]
bertforpretraining:[0.411758, 0.0876196, 0.705667, 0.224652, 0.75167,  -1.,       -0.45543,  -0.391009, -0.524803, 0.518317,...]

Sometimes the bias is big ,may be 1e-2~1e-1, while using a fine-tuned model from  chinese_L-12_H-768_A-12, as showing below:
google-research:   [0.000858, 0.355273, -0.711266, 0.258692, 1.342211, -0.072978, -0.238096, 0.288613,  -0.121792,   -0.37079, ...]
bertforpretraining:[0.017701, 0.348385, -0.742679, 0.240423, 1.337542, -0.0840113, -0.23040, 0.281977, -0.1528175, -0.3525075, ...]<|||||>@qqaatw 
ps:
hidden_states from bertforpretraining were fetched like following:
config = BertConfig.from_json_file('d:/workspace/bert-google/chinese_L-12_H-768_A-12/bert_config.json')
model = BertForPreTraining.from_pretrained('d:/workspace/bert-google/chinese_L-12_H-768_A-12/bert_model.ckpt',from_tf=True,config=config)
tokenizer = BertTokenizerFast.from_pretrained('d:/workspace/bert-google/chinese_L-12_H-768_A-12/')
inputs = tokenizer('看见我火龙果了吗',return_tensors='pt')
outputs = model(**inputs,output_hidden_states=True)
print(outputs.hidden_states[-1][0,0,:10].tolist())

And features of each output layer were fetched using extract_features.py post by google-research git repo. <|||||>Hey @bengshaoye,

```
config = BertConfig.from_json_file('d:/workspace/bert-google/chinese_L-12_H-768_A-12/bert_config.json')
```
Could you change the hidden_act from `gelu` to `gelu_new` in `bert_config.json` and try again?<|||||>Thanks a lot.
gelu_new works fine for BertForPreTraining, now they look the same with gelu in original bert and gelu_new in transformers bert.