Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+architecture.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,49 @@
+GENBIO AI COMMUNITY LICENSE AGREEMENT
+
+This GenBio AI Community License Agreement (the “License”) constitutes an agreement between you or the legal entity you represent (“you” or “your”) and GENBIO.AI, INC. (“GenBio”), governing your use of the GenBio Materials. If you are using the GenBio Materials on behalf of a legal entity, you represent and warrant to GenBio that you have full legal authority to act on behalf of that legal entity as applicable under the License. If you do not have the authority to accept this License or if you disagree with any or all of the License, you shall not use the GenBio Materials in any manner. By using or distributing any portion or element of the GenBio Materials, you imply your agreement to be bound by the License.
+
+“GenBio Materials” means any datasets, code, model weights or any other materials provided by GenBio at the following GitHub Page https://github.com/genbio-ai or Hugging Face Page https://huggingface.co/genbio-ai, including any updates or modifications made from time to time, whether in Source or Object form, and is made available to you under this License.
+
+
+1. License Grant.
+      1.1 License Scope. Subject to the terms of this License, GenBio grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable and royalty-free limited license under GenBio’s intellectual property or other rights owned by GenBio embodied in the GenBio Materials to use, reproduce, distribute, and create Derivative Works of, and make modifications to, the GenBio Materials for any Non-Commercial Purposes.
+      1.2 Use Restrictions. Restricted activities in relation to the License or use of GenBio Materials include:
+            1.2.1 You shall use the GenBio Materials, Contributions, Derivative Works, Outputs and Output Derivatives (as defined below) solely for Non-Commercial Purposes;
+            1.2.2 You shall not, directly or indirectly: (a) use or provide access to any Outputs or Output Derivatives to train, optimize, improve, or otherwise enhance the functionality or performance of any machine learning models or related technologies that are similar to the GenBio Materials; (b) engage in any form of model distillation or other methods that would achieve the purposes described in subsection (a) above. Notwithstanding the foregoing, you may use Outputs and Output Derivatives to train, optimize, improve, or enhance the functionality or performance of: (i) The GenBio Materials itself; and (ii) downstream Derivative Works of the GenBio Materials;
+            1.2.3 Your use of the GenBio Materials shall be subject to any additional terms and conditions that: (a) GenBio provides to you separately; or (b) GenBio otherwise makes available to you.
+
+2. Sharing and Distribution.  
+      2.1 Subject to Section 1, if you distribute or make available the GenBio Materials or a Derivative Work to a third party for your Non-Commercial Purposes, in Source or Object form, you shall: 
+            2.1.1 provide a copy of this License to that third party;
+            2.1.2 retain the following attribution notice within a “Notice” text file distributed as a part of such copies: “This is licensed under the GenBio AI Community License Agreement, Copyright ©  GENBIO.AI, INC. All Rights Reserved”; and
+            2.1.3 prominently display “Powered by GenBio AI” on a related website, user interface, blogpost, about page, or product documentation. 
+      2.2 If You create a Derivative Work, you may add your own attribution notice(s) to the “Notice” text file included with that Derivative Work, provided that you clearly indicate which attributions apply to the GenBio Materials and state in the “Notice” text file that you changed the GenBio Materials and how it was modified.
+
+3. Submission of Contribution.
+      Unless you explicitly state otherwise, any Contribution intentionally submitted for inclusion in the GenBio Materials by you to GenBio shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with GenBio regarding such Contributions.
+
+4. Export Control.
+      You shall comply with the applicable U.S. Foreign Corrupt Practices Act and all applicable export laws, restrictions and regulations of the U.S. Department of Commerce, and any other applicable U.S. and foreign authority.
+
+5. Disclaimer of Warranty.
+      GENBIO MATERIALS PROVIDED BY GENBIO OR ANY OUTPUT YOU RECEIVED ARE PROVIDED “AS IS.” EXCEPT TO THE EXTENT PROHIBITED BY LAW. GENBIO MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND, WHETHER EXPRESS, IMPLIED OR OTHERWISE, REGARDING THE ACCURACY, COMPLETENESS OR PERFORMANCE OF THE SERVICES AND YOUR OUTPUT, OR WITH RESPECT TO SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT. 
+
+6. Limitation of Liability.  
+      In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the GenBio Materials (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+7. General Terms. 
+      7.1 Relationship of Parties.  You and GenBio are independent contractors, and nothing herein shall be deemed to constitute either party as the agent or representative of the other or both parties as joint venturers or partners for any purpose.
+      7.2 Assignment.  This License and the rights and obligations herein may not be assigned or transferred, in whole or in part, by You without the prior written consent of GenBio. Any assignment in violation of this provision is void. GenBio may freely assign or transfer this License, in whole or in part. This License shall be binding upon, and inure to the benefit of, the successors and permitted assigns of the parties.
+      7.3 Governing Law.  This License shall be governed, construed and interpreted in accordance with the laws of the State of California, without giving effect to principles of conflicts of law. Each of the parties to this License consents to the exclusive jurisdiction and venue of the courts of the state and federal courts of California.
+      7.4 Severability. If any provision of this License is held to be invalid, illegal or unenforceable in any respect, that provision shall be limited or eliminated to the minimum extent necessary so that this License otherwise remains in full force and effect and enforceable.
+
+8. Definitions.
+      8.1 “Commercial Entity” means any entity engaged in any activity intended for or directed toward commercial advantage or monetary compensation, including, without limitation, the development of any product or service intended to be sold or made available for a fee. For the purpose of this License, references to a Commercial Entity expressly exclude any universities, non-profit organizations, not-for-profit entities, research institutes and educational and government bodies.
+      8.2 “Contribution” means any work of authorship, including the original version of the GenBio Materials and any modifications or additions to that GenBio Materials or Derivative Works thereof, that is intentionally submitted to GenBio for inclusion in the GenBio Materials by the copyright owner or by an individual or legal entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, “submitted” means any form of electronic, verbal, or written communication sent to GenBio or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GenBio for the purpose of discussing and improving the GenBio Materials, but excluding Outputs and all communications that are conspicuously marked or otherwise designated in writing by the copyright owner as “Not a Contribution”.
+      8.3 “Contributor” means GenBio and any individual or legal entity on behalf of whom a Contribution has been received by GenBio and subsequently incorporated within the GenBio Materials.
+      8.4 “Derivative Work” means any work, whether in Source or Object form, that is based on (or derived from) the GenBio Materials and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the GenBio Materials and Derivative Works thereof.
+      8.5 “Non-Commercial Purposes” means uses not intended for or directed toward commercial advantage or monetary compensation, or the facilitation of development of any product or service to be sold or made available for a fee. For the avoidance of doubt, the provision of Outputs as a service is not a Non-Commercial Purpose.
+      8.6 “Object” means any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+      8.7 “Output” means any output, including any protein sequence, structure prediction, functional annotation, molecule, descriptions of a molecule, model, sequence, text, and/or image that is elicited directly or indirectly by, or otherwise made available to, you in connection with your use of the GenBio Materials, including, but not limited to, the use of AI-Powered Technology. For the avoidance of doubt, it includes any intermediate results, such as activations across model layers, intermediate outputs from model layers (e.g., attention maps), as well as gradients and embeddings produced by the GenBio Materials.
+      8.8 “Output Derivatives” means any enhancements, modifications and derivative works of Outputs (including, but not limited to, any derivative sequences or molecules).
+      8.9 “Source” means the preferred form for making modifications, including but not limited to GenBio Materials source code, documentation source, and configuration files.

README.md

@@ -1,3 +1,161 @@
----
-license: unknown
----
+tags:
+
+- biology
+  license: other
+
+---
+
+# AIDO.RAGProtein-3B
+
+AIDO.RAGProtein-3B (AIDO.RAGPLM) is a pretrained model for Retrieval-AuGmented protein language model in an [AI-driven Digital Organism](https://arxiv.org/abs/2412.06993). AIDO.RAGProtein-3B (and [AIDO.RAGFold](https://www.biorxiv.org/content/10.1101/2024.12.02.626519v1)) integrates pre-trained protein language models with retrieved MSA, allowing for the incorporation of co-evolutionary information in structure prediction while compensating for insufficient MSA information through large-scale pretraining.
+
+AIDO.RAGProtein-3B surpasses single-sequence protein language models in perplexity, contact prediction, and fitness prediction. We utilized AIDO.RAGProtein-3B as the feature extractor for protein structure prediction, resulting in the development of [AIDO.RAGFold](https://www.biorxiv.org/content/10.1101/2024.12.02.626519v1). When sufficient MSA is available, AIDO.RAGFold achieves TM-scores comparable to AlphaFold2 and operates up to eight times faster. In scenarios where MSA is insufficient, our method significantly outperforms AlphaFold2 (∆TM-score=0.379, 0.116 and 0.059 for 0, 5 and 10 MSA sequences as input).
+
+## Model Architecture Details
+
+AIDO.RAGProtein-3B is a transformer encoder-only architecture with the dense MLP layer in each transformer block (Panel **c** below). It uses single amino acid tokenization and is optimized using a masked languange modeling (MLM) training objective.
+
+<center><img src="architecture.png" alt="An Overview of AIDO.Protein" style="width:70%; height:auto;" /></center>
+
+More architecture details are shown below:  
+
+| Model Arch         | Value |
+| ------------------ | :---: |
+| Num Attention Head |  40   |
+| Num Hidden Layer   |  36   |
+| Hidden Size        | 2560  |
+| FFN Hidden Size    | 6832  |
+| Context Length     | 12.8K |
+
+## Pre-training of AIDO.RAGProtein-3B
+
+### Data
+
+**UniRef50/Uniclust30 MSA dataset**: We utilized sequences from UniRef50 as queries to search for homologous sequences in UniClust30, subsequently constructing multiple sequence alignments (MSAs). UniRef50 comprises a total of 53.6 million sequences. Using HHblits, we searched all sequences, identifying over 25 homologous sequences for 23.7 million of them. This dataset was directly used as the training set, referred to as `HHblits_MSA`. The remaining 29.9 million sequences were input into MSA Retriever, resulting in 7.7 million sequences with more than 25 homologous sequences. This dataset was designated as `Retriever_MSA`. During training, RAGPLM randomly sampled from the two datasets with probabilities of 0.75 and 0.25
+
+### Training Details
+
+We fine-tuned a pretrained masked language model with 3-billion parameters ([MLM-3B](https://www.biorxiv.org/content/10.1101/2024.06.06.597716v1)) using MSA data by concatenating the query sequence with homologous sequences. We introduced several modifications to the standard BERT masking strategy: (1) We randomly sampled `0.05×L` span positions from a query sequence of length `L`, with span lengths following a geometric distribution (`p=0.2`), and capped the maximum length at 10. Our experiments revealed that this settings lead to an average of 15% of the query tokens were masked. (2) To prevent information leakage, when a residue was selected, all residues at the same index across all sequences (the column of the MSA matrix) were also masked. (3) When a column of MSA was selected for masking, the entire column was replaced with the `<MASK>` token in 80% of cases, with random amino acids in 10% of cases, and remained unchanged in the remaining 10% of cases. To help the model distinguish which tokens are from the same chain and which tokens have the same residue index, we use [2D rotary position embedding](https://arxiv.org/abs/2406.05347) to encode the tokens.
+
+|                  | MLM-3B             | AIDO.RAGProtein-3B         |
+| ---------------- | ------------------ | -------------------------- |
+| Training data    | UniRef+ColabFoldDB | HHblits_MSA, Retriever_MSA |
+| Initial params   | Random             | MLM-3B                     |
+| Learning rate    | 2.5e-4             | 1e-4                       |
+| Training tokens  | 1000B              | 100B                       |
+| Batch size       | 2560               | 256                        |
+| Micro batch size | 4                  | 1                          |
+| Sample length    | 1024               | 12,800                     |
+| Attention        | Bi-directional     | Bi-directional             |
+
+### Tokenization
+
+We encode protein sequence with single amino acid resolution with 44 vocabularies, where 24 tokens represent amino acid types and 20 are special tokens. Sequences were also suffixed with a `[SEP]` token as hooks for downstream tasks. 
+
+## Evaluation of AIDO.RAGProtein-3B 
+
+AIDO.RAGProtein-3B surpasses single-sequence protein language models in perplexity, contact prediction, and fitness prediction. Subsequently, we utilized AIDO.RAGProtein-3B as a feature extractor, integrating it with the folding trunks and Structure Modules to achieve end-to-end structural prediction ([AIDO.RAGFold](https://www.biorxiv.org/content/10.1101/2024.12.02.626519v1)). Our findings indicate that when sufficient MSA is available, our method achieves results comparable to AlphaFold2 and is eight times faster; when MSA is insufficient, our method significantly outperforms AlphaFold2.
+
+## Results
+
+### Unsupervised Contact Prediction
+
+<center><img src="unsupervised_contact_prediction.png" alt="xxx" style="width:70%; height:auto;" /></center>
+
+### Supervised Contact Prediction & ProteinGym zero-shot prediction
+
+<center><img src="supervised_contact_prediction_fitness.png" alt="xxx" style="width:70%; height:auto;" /></center>
+
+### AIDO.RAGFold
+
+<center><img src="structure_prediction.png" alt="xxx" style="width:70%; height:auto;" /></center>
+
+## How to Use
+
+### Build any downstream models from this backbone with ModelGenerator
+
+For more information, visit: [Model Generator](https://github.com/genbio-ai/modelgenerator)
+
+```bash
+mgen fit --model SequenceClassification --model.backbone aido_ragprotein_3b --data SequenceClassificationDataModule --data.path <hf_or_local_path_to_your_dataset>
+mgen test --model SequenceClassification --model.backbone aido_ragprotein_3b --data SequenceClassificationDataModule --data.path <hf_or_local_path_to_your_dataset>
+```
+
+### Or use directly in Python
+
+#### Embedding
+
+```python
+import torch
+from modelgenerator.tasks import Embed
+model = Embed.from_config({"model.backbone": "aido_ragprotein_3b"}).eval()
+model.backbone.max_length = 12800
+data = torch.load("ModelGenerator/experiments/AIDO.RAGPLM/examples.pt", 'cpu')[0]
+transformed_batch = model.transform(data)
+with torch.no_grad():
+    embedding = model(transformed_batch)
+
+print(embedding.shape)
+```
+
+#### Sequence Level Classification
+
+```python
+import torch
+from modelgenerator.tasks import SequenceClassification
+model = SequenceClassification.from_config({"model.backbone": "aido_ragprotein_3b", "model.n_classes": 2}).eval()
+model.backbone.max_length = 12800
+data = torch.load("ModelGenerator/experiments/AIDO.RAGPLM/examples.pt", 'cpu')[0]
+transformed_batch = model.transform(data)
+with torch.no_grad():
+    logits = model(transformed_batch)
+
+print(logits)
+print(torch.argmax(logits, dim=-1))
+```
+
+#### Token Level Classification
+
+```python
+import torch
+from modelgenerator.tasks import TokenClassification
+model = TokenClassification.from_config({"model.backbone": "aido_ragprotein_3b", "model.n_classes": 3}).eval()
+model.backbone.max_length = 12800
+data = torch.load("ModelGenerator/experiments/AIDO.RAGPLM/examples.pt", 'cpu')[0]
+transformed_batch = model.transform(data)
+with torch.no_grad():
+    logits = model(transformed_batch)
+
+print(logits)
+print(torch.argmax(logits, dim=-1))
+```
+
+#### Regression
+
+```python
+from modelgenerator.tasks import SequenceRegression
+model = SequenceRegression.from_config({"model.backbone": "aido_protein_16b_ragplm"}).eval()
+model.backbone.max_length = 12800
+data = torch.load("experiments/AIDO.RAGPLM/examples.pt", 'cpu')[0]
+transformed_batch = model.transform(data)
+with torch.no_grad():
+    logits = model(transformed_batch)
+
+print(logits.shape)
+```
+
+# Citation
+
+Please cite AIDO.RAGProtein-3B using the following BibTex code:
+
+```
+@article {Li2024.12.02.626519,
+    author = {Li, Pan and Cheng, Xingyi and Song, Le and Xing, Eric},
+    title = {Retrieval Augmented Protein Language Models for Protein Structure Prediction},
+    url = {https://www.biorxiv.org/content/10.1101/2024.12.02.626519v1},
+    year = {2024},
+    doi = {10.1101/2024.12.02.626519},
+    publisher = {bioRxiv},
+    booktitle={NeurIPS 2024 Workshop on Machine Learning in Structural Biology},
+}
+```

config.json

@@ -0,0 +1,37 @@
+{
+  "add_linear_bias": true,
+  "apply_residual_connection_post_layernorm": true,
+  "architectures": [
+    "FM4BioForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0,
+  "experts_per_token": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "geglu",
+  "hidden_dropout_prob": 0,
+  "hidden_size": 2560,
+  "initializer_range": 0.02,
+  "intermediate_size": 6832,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 2048,
+  "model_type": "fm4bio",
+  "moe": false,
+  "normalization_type": "LayerNorm",
+  "num_attention_heads": 40,
+  "num_experts": 8,
+  "num_hidden_layers": 36,
+  "output_vocab_size": null,
+  "pad_token_id": 0,
+  "position_embedding_type": "rope_2d",
+  "rotary_percent": 1.0,
+  "seq_len_interpolation_factor": null,
+  "str_embedding_in": null,
+  "tie_word_embeddings": false,
+  "tokenizer_insert_str_tokens": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "use_lm_head": false,
+  "vocab_size": 128
+}

generation_config.json

@@ -0,0 +1,5 @@
+{
+  "_from_model_config": true,
+  "pad_token_id": 0,
+  "transformers_version": "4.48.3"
+}

pytorch_model-00001-of-00003.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d50efa37938af31911a03e8e7fef8bed5f0c269a007020a13c5e699b8bd8628a
+size 4969563679

pytorch_model-00002-of-00003.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:809189cd3742ca97ce207a6ca924782001ce8dcf2db6d771bdf1036495f176af
+size 4898248668

pytorch_model-00003-of-00003.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:433f42e6287f0644e514038cd02454a36a223093ef5fd0cb15bd335eb456cf66
+size 1470833384

pytorch_model.bin.index.json

@@ -0,0 +1,587 @@
+{
+  "metadata": {
+    "total_size": 11338446336
+  },
+  "weight_map": {
+    "bert.embeddings.word_embeddings.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.10.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.11.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.12.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.13.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.14.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.15.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.15.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.15.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.16.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.16.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.17.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.18.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.19.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.2.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.20.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.20.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.21.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.22.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.23.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.24.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.25.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.26.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.27.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.28.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.29.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.3.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.3.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.30.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.mlp.dense_4h_to_h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.mlp.dense_h_to_4h.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.30.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.output.dense.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.output.dense.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.key.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.key.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.query.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.query.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.value.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.attention.self.value.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.ln.bias": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.ln.weight": "pytorch_model-00002-of-00003.bin",
+    "bert.encoder.layer.31.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.31.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.31.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.31.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.output.dense.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.output.dense.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.key.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.key.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.query.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.query.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.value.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.attention.self.value.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.32.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.output.dense.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.output.dense.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.key.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.key.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.query.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.query.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.value.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.attention.self.value.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.33.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.output.dense.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.output.dense.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.key.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.key.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.query.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.query.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.value.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.attention.self.value.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.34.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.output.dense.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.output.dense.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.key.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.key.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.query.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.query.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.value.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.attention.self.value.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.mlp.dense_4h_to_h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.mlp.dense_h_to_4h.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.35.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.layer.4.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.4.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.5.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.6.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.7.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.8.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.output.dense.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.output.dense.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.key.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.key.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.query.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.query.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.value.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.attention.self.value.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.ln.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.ln.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.mlp.dense_4h_to_h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.mlp.dense_h_to_4h.bias": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.layer.9.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00003.bin",
+    "bert.encoder.ln.bias": "pytorch_model-00003-of-00003.bin",
+    "bert.encoder.ln.weight": "pytorch_model-00003-of-00003.bin",
+    "output_embed.weight": "pytorch_model-00003-of-00003.bin"
+  }
+}