bigdefence commited on
Commit
6cb2945
·
verified ·
1 Parent(s): 9a71c37

Delete checkpoint-6981

Browse files
checkpoint-6981/README.md DELETED
@@ -1,207 +0,0 @@
1
- ---
2
- base_model: ./hf_hub/EXAONE
3
- library_name: peft
4
- pipeline_tag: text-generation
5
- tags:
6
- - base_model:adapter:./hf_hub/EXAONE
7
- - lora
8
- - transformers
9
- ---
10
-
11
- # Model Card for Model ID
12
-
13
- <!-- Provide a quick summary of what the model is/does. -->
14
-
15
-
16
-
17
- ## Model Details
18
-
19
- ### Model Description
20
-
21
- <!-- Provide a longer summary of what this model is. -->
22
-
23
-
24
-
25
- - **Developed by:** [More Information Needed]
26
- - **Funded by [optional]:** [More Information Needed]
27
- - **Shared by [optional]:** [More Information Needed]
28
- - **Model type:** [More Information Needed]
29
- - **Language(s) (NLP):** [More Information Needed]
30
- - **License:** [More Information Needed]
31
- - **Finetuned from model [optional]:** [More Information Needed]
32
-
33
- ### Model Sources [optional]
34
-
35
- <!-- Provide the basic links for the model. -->
36
-
37
- - **Repository:** [More Information Needed]
38
- - **Paper [optional]:** [More Information Needed]
39
- - **Demo [optional]:** [More Information Needed]
40
-
41
- ## Uses
42
-
43
- <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
-
45
- ### Direct Use
46
-
47
- <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
-
49
- [More Information Needed]
50
-
51
- ### Downstream Use [optional]
52
-
53
- <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
-
55
- [More Information Needed]
56
-
57
- ### Out-of-Scope Use
58
-
59
- <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
-
61
- [More Information Needed]
62
-
63
- ## Bias, Risks, and Limitations
64
-
65
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
-
67
- [More Information Needed]
68
-
69
- ### Recommendations
70
-
71
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
-
73
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
-
75
- ## How to Get Started with the Model
76
-
77
- Use the code below to get started with the model.
78
-
79
- [More Information Needed]
80
-
81
- ## Training Details
82
-
83
- ### Training Data
84
-
85
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
-
87
- [More Information Needed]
88
-
89
- ### Training Procedure
90
-
91
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
-
93
- #### Preprocessing [optional]
94
-
95
- [More Information Needed]
96
-
97
-
98
- #### Training Hyperparameters
99
-
100
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
-
102
- #### Speeds, Sizes, Times [optional]
103
-
104
- <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
-
106
- [More Information Needed]
107
-
108
- ## Evaluation
109
-
110
- <!-- This section describes the evaluation protocols and provides the results. -->
111
-
112
- ### Testing Data, Factors & Metrics
113
-
114
- #### Testing Data
115
-
116
- <!-- This should link to a Dataset Card if possible. -->
117
-
118
- [More Information Needed]
119
-
120
- #### Factors
121
-
122
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
-
124
- [More Information Needed]
125
-
126
- #### Metrics
127
-
128
- <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
-
130
- [More Information Needed]
131
-
132
- ### Results
133
-
134
- [More Information Needed]
135
-
136
- #### Summary
137
-
138
-
139
-
140
- ## Model Examination [optional]
141
-
142
- <!-- Relevant interpretability work for the model goes here -->
143
-
144
- [More Information Needed]
145
-
146
- ## Environmental Impact
147
-
148
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
-
150
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
-
152
- - **Hardware Type:** [More Information Needed]
153
- - **Hours used:** [More Information Needed]
154
- - **Cloud Provider:** [More Information Needed]
155
- - **Compute Region:** [More Information Needed]
156
- - **Carbon Emitted:** [More Information Needed]
157
-
158
- ## Technical Specifications [optional]
159
-
160
- ### Model Architecture and Objective
161
-
162
- [More Information Needed]
163
-
164
- ### Compute Infrastructure
165
-
166
- [More Information Needed]
167
-
168
- #### Hardware
169
-
170
- [More Information Needed]
171
-
172
- #### Software
173
-
174
- [More Information Needed]
175
-
176
- ## Citation [optional]
177
-
178
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
-
180
- **BibTeX:**
181
-
182
- [More Information Needed]
183
-
184
- **APA:**
185
-
186
- [More Information Needed]
187
-
188
- ## Glossary [optional]
189
-
190
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
-
192
- [More Information Needed]
193
-
194
- ## More Information [optional]
195
-
196
- [More Information Needed]
197
-
198
- ## Model Card Authors [optional]
199
-
200
- [More Information Needed]
201
-
202
- ## Model Card Contact
203
-
204
- [More Information Needed]
205
- ### Framework versions
206
-
207
- - PEFT 0.17.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoint-6981/adapter_config.json DELETED
@@ -1,129 +0,0 @@
1
- {
2
- "alpha_pattern": {},
3
- "auto_mapping": null,
4
- "base_model_name_or_path": "./hf_hub/EXAONE",
5
- "bias": "none",
6
- "corda_config": null,
7
- "eva_config": null,
8
- "exclude_modules": null,
9
- "fan_in_fan_out": false,
10
- "inference_mode": true,
11
- "init_lora_weights": true,
12
- "layer_replication": null,
13
- "layers_pattern": null,
14
- "layers_to_transform": null,
15
- "loftq_config": {},
16
- "lora_alpha": 16,
17
- "lora_bias": false,
18
- "lora_dropout": 0.05,
19
- "megatron_config": null,
20
- "megatron_core": "megatron.core",
21
- "modules_to_save": null,
22
- "peft_type": "LORA",
23
- "qalora_group_size": 16,
24
- "r": 8,
25
- "rank_pattern": {},
26
- "revision": null,
27
- "target_modules": [
28
- "model.layers.2.self_attn.v_proj",
29
- "model.layers.10.self_attn.q_proj",
30
- "model.layers.8.self_attn.v_proj",
31
- "model.layers.29.self_attn.k_proj",
32
- "model.layers.13.self_attn.k_proj",
33
- "model.layers.19.self_attn.q_proj",
34
- "model.layers.14.self_attn.k_proj",
35
- "model.layers.29.self_attn.v_proj",
36
- "model.layers.15.self_attn.v_proj",
37
- "model.layers.18.self_attn.v_proj",
38
- "model.layers.4.self_attn.k_proj",
39
- "model.layers.28.self_attn.k_proj",
40
- "model.layers.6.self_attn.v_proj",
41
- "model.layers.25.self_attn.v_proj",
42
- "model.layers.27.self_attn.k_proj",
43
- "model.layers.14.self_attn.q_proj",
44
- "model.layers.26.self_attn.v_proj",
45
- "model.layers.11.self_attn.v_proj",
46
- "model.layers.18.self_attn.q_proj",
47
- "model.layers.19.self_attn.k_proj",
48
- "model.layers.27.self_attn.v_proj",
49
- "model.layers.0.self_attn.q_proj",
50
- "model.layers.17.self_attn.k_proj",
51
- "model.layers.26.self_attn.k_proj",
52
- "model.layers.10.self_attn.v_proj",
53
- "model.layers.4.self_attn.q_proj",
54
- "model.layers.11.self_attn.k_proj",
55
- "model.layers.5.self_attn.v_proj",
56
- "model.layers.21.self_attn.v_proj",
57
- "model.layers.23.self_attn.q_proj",
58
- "model.layers.29.self_attn.q_proj",
59
- "up_proj",
60
- "model.layers.20.self_attn.k_proj",
61
- "model.layers.16.self_attn.q_proj",
62
- "model.layers.11.self_attn.q_proj",
63
- "model.layers.7.self_attn.q_proj",
64
- "model.layers.17.self_attn.q_proj",
65
- "model.layers.9.self_attn.v_proj",
66
- "model.layers.27.self_attn.q_proj",
67
- "model.layers.4.self_attn.v_proj",
68
- "model.layers.21.self_attn.k_proj",
69
- "model.layers.20.self_attn.v_proj",
70
- "model.layers.28.self_attn.q_proj",
71
- "model.layers.16.self_attn.k_proj",
72
- "model.layers.12.self_attn.v_proj",
73
- "model.layers.6.self_attn.q_proj",
74
- "model.layers.1.self_attn.v_proj",
75
- "model.layers.20.self_attn.q_proj",
76
- "model.layers.0.self_attn.v_proj",
77
- "model.layers.22.self_attn.v_proj",
78
- "model.layers.2.self_attn.q_proj",
79
- "model.layers.1.self_attn.q_proj",
80
- "model.layers.25.self_attn.q_proj",
81
- "model.layers.1.self_attn.k_proj",
82
- "model.layers.6.self_attn.k_proj",
83
- "model.layers.9.self_attn.q_proj",
84
- "model.layers.18.self_attn.k_proj",
85
- "model.layers.26.self_attn.q_proj",
86
- "model.layers.15.self_attn.k_proj",
87
- "model.layers.0.self_attn.k_proj",
88
- "model.layers.14.self_attn.v_proj",
89
- "model.layers.25.self_attn.k_proj",
90
- "model.layers.22.self_attn.k_proj",
91
- "model.layers.23.self_attn.v_proj",
92
- "model.layers.7.self_attn.v_proj",
93
- "gate_proj",
94
- "model.layers.15.self_attn.q_proj",
95
- "model.layers.21.self_attn.q_proj",
96
- "model.layers.23.self_attn.k_proj",
97
- "model.layers.10.self_attn.k_proj",
98
- "model.layers.3.self_attn.k_proj",
99
- "model.layers.12.self_attn.k_proj",
100
- "model.layers.28.self_attn.v_proj",
101
- "model.layers.24.self_attn.k_proj",
102
- "model.layers.17.self_attn.v_proj",
103
- "model.layers.7.self_attn.k_proj",
104
- "model.layers.12.self_attn.q_proj",
105
- "model.layers.13.self_attn.v_proj",
106
- "model.layers.13.self_attn.q_proj",
107
- "model.layers.3.self_attn.v_proj",
108
- "model.layers.5.self_attn.q_proj",
109
- "model.layers.5.self_attn.k_proj",
110
- "model.layers.24.self_attn.v_proj",
111
- "model.layers.3.self_attn.q_proj",
112
- "model.layers.8.self_attn.k_proj",
113
- "model.layers.9.self_attn.k_proj",
114
- "model.layers.8.self_attn.q_proj",
115
- "model.layers.22.self_attn.q_proj",
116
- "model.layers.2.self_attn.k_proj",
117
- "model.layers.24.self_attn.q_proj",
118
- "o_proj",
119
- "down_proj",
120
- "model.layers.16.self_attn.v_proj",
121
- "model.layers.19.self_attn.v_proj"
122
- ],
123
- "target_parameters": null,
124
- "task_type": "CAUSAL_LM",
125
- "trainable_token_indices": null,
126
- "use_dora": false,
127
- "use_qalora": false,
128
- "use_rslora": false
129
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoint-6981/adapter_model.safetensors DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:122af47cb8bcee373e2440191c1e9112344a4e711c68cea7ed40b20b9152678e
3
- size 15292392
 
 
 
 
checkpoint-6981/chat_template.jinja DELETED
@@ -1,146 +0,0 @@
1
- {%- if not skip_think is defined %}
2
- {%- set skip_think = true %}
3
- {%- endif %}
4
-
5
- {%- set role_indicators = {
6
- 'user': '[|user|]\n',
7
- 'assistant': '[|assistant|]\n',
8
- 'system': '[|system|]\n',
9
- 'tool': '[|tool|]\n'
10
- } %}
11
- {%- set end_of_turn = '[|endofturn|]\n' %}
12
-
13
-
14
- {%- macro available_tools(tools) %}
15
- {{- "# Available Tools" }}
16
- {{- "\nYou can use none, one, or multiple of the following tools by calling them as functions to help with the user’s query." }}
17
- {{- "\nHere are the tools available to you in JSON format within <tool> and </tool> tags:\n" }}
18
- {%- for tool in tools %}
19
- {{- "<tool>" }}
20
- {{- tool | tojson(ensure_ascii=False) | safe }}
21
- {{- "</tool>\n" }}
22
- {%- endfor %}
23
-
24
- {{- "\nFor each function call you want to make, return a JSON object with function name and arguments within <tool_call> and </tool_call> tags, like:" }}
25
- {{- "\n<tool_call>{\"name\": function_1_name, \"arguments\": {argument_1_name: argument_1_value, argument_2_name: argument_2_value}}</tool_call>" }}
26
- {{- "\n<tool_call>{\"name\": function_2_name, \"arguments\": {...}}</tool_call>\n..." }}
27
- {{- "\nNote that if no argument name is specified for a tool, you can just print the argument value directly, without the argument name or JSON formatting." }}
28
- {%- endmacro %}
29
-
30
-
31
- {%- set ns = namespace(last_query_index = messages|length - 1) %}
32
- {%- for message in messages %}
33
- {%- if message.role == "user" and message.content is string %}
34
- {%- set ns.last_query_index = loop.index0 -%}
35
- {%- endif %}
36
- {%- endfor %}
37
-
38
- {%- for i in range(messages | length) %}
39
- {%- set msg = messages[i] %}
40
- {%- set role = msg.role %}
41
- {%- if role not in role_indicators %}
42
- {{- raise_exception('Unknown role: ' ~ role) }}
43
- {%- endif %}
44
-
45
- {%- if i == 0 %}
46
- {%- if role == 'system' %}
47
- {{- role_indicators['system'] }}
48
- {{- msg.content }}
49
- {%- if tools is defined and tools %}
50
- {{- "\n\n" }}{{- available_tools(tools) }}
51
- {%- endif %}
52
- {{- end_of_turn -}}
53
- {%- continue %}
54
- {%- elif tools is defined and tools %}
55
- {{- role_indicators['system'] }}
56
- {{- available_tools(tools) }}
57
- {{- end_of_turn -}}
58
- {%- endif %}
59
- {%- endif %}
60
-
61
- {%- if role == 'assistant' %}
62
- {{- role_indicators['assistant'] }}
63
-
64
- {%- if msg.content %}
65
- {%- if "</think>" in msg.content %}
66
- {%- set content = msg.content.split('</think>')[-1].strip() %}
67
- {%- set reasoning_content = msg.content.split('</think>')[0].strip() %}
68
- {%- if reasoning_content.startswith("<think>") %}
69
- {%- set reasoning_content = reasoning_content[9:].strip() %}
70
- {%- endif %}
71
- {%- else %}
72
- {%- set content = msg.content %}
73
- {%- endif %}
74
-
75
- {%- if msg.reasoning_content %}
76
- {%- set reasoning_content = msg.reasoning_content %}
77
- {%- endif %}
78
-
79
- {%- if (not skip_think and loop.last) and reasoning_content is defined %}
80
- {{- "<think>\n" }}
81
- {{- reasoning_content}}
82
- {{- "\n</think>\n\n" }}
83
- {%- else %}
84
- {{- "<think>\n\n</think>\n\n" }}
85
- {%- endif %}
86
- {{- content }}
87
- {%- endif %}
88
-
89
- {%- if msg.tool_calls %}
90
- {%- if msg.content %}
91
- {{- "\n" }}
92
- {%- else %}
93
- {{- "<think>\n\n</think>\n\n" }}
94
- {%- endif %}
95
- {%- for tool_call in msg.tool_calls %}
96
- {%- if tool_call.function is defined %}
97
- {%- set tool_call = tool_call.function %}
98
- {%- endif %}
99
-
100
- {%- if tool_call.arguments is defined %}
101
- {%- set arguments = tool_call.arguments %}
102
- {%- elif tool_call.parameters is defined %}
103
- {%- set arguments = tool_call.parameters %}
104
- {%- else %}
105
- {{- raise_exception('arguments or parameters are mandatory: ' ~ tool_call) }}
106
- {%- endif %}
107
-
108
- {{- "<tool_call>" }}{"name": "{{- tool_call.name }}", "arguments": {{ arguments | tojson(ensure_ascii=False) | safe }}}{{- "</tool_call>" }}
109
-
110
- {%- if not loop.last %}
111
- {{- "\n" }}
112
- {%- endif %}
113
-
114
- {%- endfor %}
115
- {%- endif %}
116
- {{- end_of_turn -}}
117
-
118
- {%- elif role == "tool" %}
119
- {%- if i == 0 or messages[i - 1].role != "tool" %}
120
- {{- role_indicators['tool'] }}
121
- {%- endif %}
122
- {%- if msg.content is defined %}
123
- {{- "<tool_result>" }}{"result": {{ msg.content | tojson(ensure_ascii=False) | safe }}}{{- "</tool_result>" }}
124
- {%- endif %}
125
- {%- if loop.last or messages[i + 1].role != "tool" %}
126
- {{- end_of_turn -}}
127
- {%- else %}
128
- {{- "\n" }}
129
- {%- endif %}
130
-
131
- {%- else %}
132
- {{- role_indicators[role] }}
133
- {{- msg.content }}
134
- {{- end_of_turn -}}
135
- {%- endif %}
136
- {% endfor %}
137
-
138
-
139
- {%- if add_generation_prompt %}
140
- {{- role_indicators['assistant'] }}
141
- {%- if enable_thinking is defined and enable_thinking is true %}
142
- {{- "<think>\n" }}
143
- {%- else %}
144
- {{- "<think>\n\n</think>\n\n" }}
145
- {%- endif %}
146
- {%- endif %}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoint-6981/global_step6981/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:45bef70e06d77d2cdd0eb2919faeed139d69df09943b2ddd7ee3083676ea4af6
3
- size 299141253
 
 
 
 
checkpoint-6981/global_step6981/mp_rank_00_model_states.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:eecc85612ab19efd4f2e6b8a087252cf2af0adc3295d6a3925b045640c41a166
3
- size 469638833
 
 
 
 
checkpoint-6981/latest DELETED
@@ -1 +0,0 @@
1
- global_step6981
 
 
checkpoint-6981/merges.txt DELETED
The diff for this file is too large to render. See raw diff
 
checkpoint-6981/rng_state.pth DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:b19995b1eb2f1692367a0ba66378c06e12ec38a1a3bf85e5ac814abec0c24a7c
3
- size 14709
 
 
 
 
checkpoint-6981/scheduler.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:12f9a2f3627a2b9b2baa775951734a34f22756727f3bb2c4aabd56591c31af48
3
- size 1465
 
 
 
 
checkpoint-6981/special_tokens_map.json DELETED
@@ -1,330 +0,0 @@
1
- {
2
- "additional_special_tokens": [
3
- "[unused0]",
4
- "[unused1]",
5
- "[unused2]",
6
- "[unused3]",
7
- "[unused4]",
8
- "[unused5]",
9
- "[unused6]",
10
- "[unused7]",
11
- "[unused8]",
12
- "[unused9]",
13
- "[unused10]",
14
- "[unused11]",
15
- "[unused12]",
16
- "[unused13]",
17
- "[unused14]",
18
- "[unused15]",
19
- "[unused16]",
20
- "[unused17]",
21
- "[unused18]",
22
- "[unused19]",
23
- "[unused20]",
24
- "[unused21]",
25
- "[unused22]",
26
- "[unused23]",
27
- "[unused24]",
28
- "[unused25]",
29
- "[unused26]",
30
- "[unused27]",
31
- "[unused28]",
32
- "[unused29]",
33
- "[unused30]",
34
- "[unused31]",
35
- "[unused32]",
36
- "[unused33]",
37
- "[unused34]",
38
- "[unused35]",
39
- "[unused36]",
40
- "[unused37]",
41
- "[unused38]",
42
- "[unused39]",
43
- "[unused40]",
44
- "[unused41]",
45
- "[unused42]",
46
- "[unused43]",
47
- "[unused44]",
48
- "[unused45]",
49
- "[unused46]",
50
- "[unused47]",
51
- "[unused48]",
52
- "[unused49]",
53
- "[unused50]",
54
- "[unused51]",
55
- "[unused52]",
56
- "[unused53]",
57
- "[unused54]",
58
- "[unused55]",
59
- "[unused56]",
60
- "[unused57]",
61
- "[unused58]",
62
- "[unused59]",
63
- "[unused60]",
64
- "[unused61]",
65
- "[unused62]",
66
- "[unused63]",
67
- "[unused64]",
68
- "[unused65]",
69
- "[unused66]",
70
- "[unused67]",
71
- "[unused68]",
72
- "[unused69]",
73
- "[unused70]",
74
- "[unused71]",
75
- "[unused72]",
76
- "[unused73]",
77
- "[unused74]",
78
- "[unused75]",
79
- "[unused76]",
80
- "[unused77]",
81
- "[unused78]",
82
- "[unused79]",
83
- "[unused80]",
84
- "[unused81]",
85
- "[unused82]",
86
- "[unused83]",
87
- "[unused84]",
88
- "[unused85]",
89
- "[unused86]",
90
- "[unused87]",
91
- "[unused88]",
92
- "[unused89]",
93
- "[unused90]",
94
- "[unused91]",
95
- "[unused92]",
96
- "[unused93]",
97
- "[unused94]",
98
- "[unused95]",
99
- "[unused96]",
100
- "[unused97]",
101
- "[unused98]",
102
- "[unused99]",
103
- "[extra_id_0]",
104
- "[extra_id_1]",
105
- "[extra_id_2]",
106
- "[extra_id_3]",
107
- "[extra_id_4]",
108
- "[extra_id_5]",
109
- "[extra_id_6]",
110
- "[extra_id_7]",
111
- "[extra_id_8]",
112
- "[extra_id_9]",
113
- "[extra_id_10]",
114
- "[extra_id_11]",
115
- "[extra_id_12]",
116
- "[extra_id_13]",
117
- "[extra_id_14]",
118
- "[extra_id_15]",
119
- "[extra_id_16]",
120
- "[extra_id_17]",
121
- "[extra_id_18]",
122
- "[extra_id_19]",
123
- "[extra_id_20]",
124
- "[extra_id_21]",
125
- "[extra_id_22]",
126
- "[extra_id_23]",
127
- "[extra_id_24]",
128
- "[extra_id_25]",
129
- "[extra_id_26]",
130
- "[extra_id_27]",
131
- "[extra_id_28]",
132
- "[extra_id_29]",
133
- "[extra_id_30]",
134
- "[extra_id_31]",
135
- "[extra_id_32]",
136
- "[extra_id_33]",
137
- "[extra_id_34]",
138
- "[extra_id_35]",
139
- "[extra_id_36]",
140
- "[extra_id_37]",
141
- "[extra_id_38]",
142
- "[extra_id_39]",
143
- "[extra_id_40]",
144
- "[extra_id_41]",
145
- "[extra_id_42]",
146
- "[extra_id_43]",
147
- "[extra_id_44]",
148
- "[extra_id_45]",
149
- "[extra_id_46]",
150
- "[extra_id_47]",
151
- "[extra_id_48]",
152
- "[extra_id_49]",
153
- "[extra_id_50]",
154
- "[extra_id_51]",
155
- "[extra_id_52]",
156
- "[extra_id_53]",
157
- "[extra_id_54]",
158
- "[extra_id_55]",
159
- "[extra_id_56]",
160
- "[extra_id_57]",
161
- "[extra_id_58]",
162
- "[extra_id_59]",
163
- "[extra_id_60]",
164
- "[extra_id_61]",
165
- "[extra_id_62]",
166
- "[extra_id_63]",
167
- "[extra_id_64]",
168
- "[extra_id_65]",
169
- "[extra_id_66]",
170
- "[extra_id_67]",
171
- "[extra_id_68]",
172
- "[extra_id_69]",
173
- "[extra_id_70]",
174
- "[extra_id_71]",
175
- "[extra_id_72]",
176
- "[extra_id_73]",
177
- "[extra_id_74]",
178
- "[extra_id_75]",
179
- "[extra_id_76]",
180
- "[extra_id_77]",
181
- "[extra_id_78]",
182
- "[extra_id_79]",
183
- "[extra_id_80]",
184
- "[extra_id_81]",
185
- "[extra_id_82]",
186
- "[extra_id_83]",
187
- "[extra_id_84]",
188
- "[extra_id_85]",
189
- "[extra_id_86]",
190
- "[extra_id_87]",
191
- "[extra_id_88]",
192
- "[extra_id_89]",
193
- "[extra_id_90]",
194
- "[extra_id_91]",
195
- "[extra_id_92]",
196
- "[extra_id_93]",
197
- "[extra_id_94]",
198
- "[extra_id_95]",
199
- "[extra_id_96]",
200
- "[extra_id_97]",
201
- "[extra_id_98]",
202
- "[extra_id_99]",
203
- "[extra_id_100]",
204
- "[extra_id_101]",
205
- "[extra_id_102]",
206
- "[extra_id_103]",
207
- "[extra_id_104]",
208
- "[extra_id_105]",
209
- "[extra_id_106]",
210
- "[extra_id_107]",
211
- "[extra_id_108]",
212
- "[extra_id_109]",
213
- "[extra_id_110]",
214
- "[extra_id_111]",
215
- "[extra_id_112]",
216
- "[extra_id_113]",
217
- "[extra_id_114]",
218
- "[extra_id_115]",
219
- "[extra_id_116]",
220
- "[extra_id_117]",
221
- "[extra_id_118]",
222
- "[extra_id_119]",
223
- "[extra_id_120]",
224
- "[extra_id_121]",
225
- "[extra_id_122]",
226
- "[extra_id_123]",
227
- "[extra_id_124]",
228
- "[extra_id_125]",
229
- "[extra_id_126]",
230
- "[extra_id_127]",
231
- "[extra_id_128]",
232
- "[extra_id_129]",
233
- "[extra_id_130]",
234
- "[extra_id_131]",
235
- "[extra_id_132]",
236
- "[extra_id_133]",
237
- "[extra_id_134]",
238
- "[extra_id_135]",
239
- "[extra_id_136]",
240
- "[extra_id_137]",
241
- "[extra_id_138]",
242
- "[extra_id_139]",
243
- "[extra_id_140]",
244
- "[extra_id_141]",
245
- "[extra_id_142]",
246
- "[extra_id_143]",
247
- "[extra_id_144]",
248
- "[extra_id_145]",
249
- "[extra_id_146]",
250
- "[extra_id_147]",
251
- "[extra_id_148]",
252
- "[extra_id_149]",
253
- "[extra_id_150]",
254
- "[extra_id_151]",
255
- "[extra_id_152]",
256
- "[extra_id_153]",
257
- "[extra_id_154]",
258
- "[extra_id_155]",
259
- "[extra_id_156]",
260
- "[extra_id_157]",
261
- "[extra_id_158]",
262
- "[extra_id_159]",
263
- "[extra_id_160]",
264
- "[extra_id_161]",
265
- "[extra_id_162]",
266
- "[extra_id_163]",
267
- "[extra_id_164]",
268
- "[extra_id_165]",
269
- "[extra_id_166]",
270
- "[extra_id_167]",
271
- "[extra_id_168]",
272
- "[extra_id_169]",
273
- "[extra_id_170]",
274
- "[extra_id_171]",
275
- "[extra_id_172]",
276
- "[extra_id_173]",
277
- "[extra_id_174]",
278
- "[extra_id_175]",
279
- "[extra_id_176]",
280
- "[extra_id_177]",
281
- "[extra_id_178]",
282
- "[extra_id_179]",
283
- "[extra_id_180]",
284
- "[extra_id_181]",
285
- "[extra_id_182]",
286
- "[extra_id_183]",
287
- "[extra_id_184]",
288
- "[extra_id_185]",
289
- "[extra_id_186]",
290
- "[extra_id_187]",
291
- "[extra_id_188]",
292
- "[|system|]",
293
- "[|tool|]",
294
- "[|assistant|]",
295
- "[|user|]",
296
- "[|endofturn|]",
297
- "PI:URL",
298
- "PI:EMAIL",
299
- "PI:ACCOUNT_NUM",
300
- "PI:PHONE_NUM",
301
- "PI:BUSINESS_NUM",
302
- "PI:ANNON",
303
- "PI:KEY",
304
- "PI:ID",
305
- "PI:IP_ADDRESS",
306
- "PI:USER"
307
- ],
308
- "bos_token": {
309
- "content": "[BOS]",
310
- "lstrip": false,
311
- "normalized": false,
312
- "rstrip": false,
313
- "single_word": false
314
- },
315
- "eos_token": {
316
- "content": "[|endofturn|]",
317
- "lstrip": false,
318
- "normalized": false,
319
- "rstrip": false,
320
- "single_word": false
321
- },
322
- "pad_token": "[PAD]",
323
- "unk_token": {
324
- "content": "[UNK]",
325
- "lstrip": false,
326
- "normalized": false,
327
- "rstrip": false,
328
- "single_word": false
329
- }
330
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoint-6981/tokenizer.json DELETED
The diff for this file is too large to render. See raw diff
 
checkpoint-6981/tokenizer_config.json DELETED
@@ -1,3219 +0,0 @@
1
- {
2
- "add_bos_token": false,
3
- "add_prefix_space": false,
4
- "added_tokens_decoder": {
5
- "0": {
6
- "content": "[PAD]",
7
- "lstrip": false,
8
- "normalized": false,
9
- "rstrip": false,
10
- "single_word": false,
11
- "special": true
12
- },
13
- "1": {
14
- "content": "[BOS]",
15
- "lstrip": false,
16
- "normalized": false,
17
- "rstrip": false,
18
- "single_word": false,
19
- "special": true
20
- },
21
- "2": {
22
- "content": "[EOS]",
23
- "lstrip": false,
24
- "normalized": false,
25
- "rstrip": false,
26
- "single_word": false,
27
- "special": true
28
- },
29
- "3": {
30
- "content": "[UNK]",
31
- "lstrip": false,
32
- "normalized": false,
33
- "rstrip": false,
34
- "single_word": false,
35
- "special": true
36
- },
37
- "4": {
38
- "content": " ",
39
- "lstrip": false,
40
- "normalized": false,
41
- "rstrip": false,
42
- "single_word": false,
43
- "special": false
44
- },
45
- "5": {
46
- "content": " ",
47
- "lstrip": false,
48
- "normalized": false,
49
- "rstrip": false,
50
- "single_word": false,
51
- "special": false
52
- },
53
- "6": {
54
- "content": " ",
55
- "lstrip": false,
56
- "normalized": false,
57
- "rstrip": false,
58
- "single_word": false,
59
- "special": false
60
- },
61
- "7": {
62
- "content": " ",
63
- "lstrip": false,
64
- "normalized": false,
65
- "rstrip": false,
66
- "single_word": false,
67
- "special": false
68
- },
69
- "8": {
70
- "content": " ",
71
- "lstrip": false,
72
- "normalized": false,
73
- "rstrip": false,
74
- "single_word": false,
75
- "special": false
76
- },
77
- "9": {
78
- "content": " ",
79
- "lstrip": false,
80
- "normalized": false,
81
- "rstrip": false,
82
- "single_word": false,
83
- "special": false
84
- },
85
- "10": {
86
- "content": " ",
87
- "lstrip": false,
88
- "normalized": false,
89
- "rstrip": false,
90
- "single_word": false,
91
- "special": false
92
- },
93
- "11": {
94
- "content": " ",
95
- "lstrip": false,
96
- "normalized": false,
97
- "rstrip": false,
98
- "single_word": false,
99
- "special": false
100
- },
101
- "12": {
102
- "content": " ",
103
- "lstrip": false,
104
- "normalized": false,
105
- "rstrip": false,
106
- "single_word": false,
107
- "special": false
108
- },
109
- "13": {
110
- "content": " ",
111
- "lstrip": false,
112
- "normalized": false,
113
- "rstrip": false,
114
- "single_word": false,
115
- "special": false
116
- },
117
- "14": {
118
- "content": " ",
119
- "lstrip": false,
120
- "normalized": false,
121
- "rstrip": false,
122
- "single_word": false,
123
- "special": false
124
- },
125
- "15": {
126
- "content": " ",
127
- "lstrip": false,
128
- "normalized": false,
129
- "rstrip": false,
130
- "single_word": false,
131
- "special": false
132
- },
133
- "16": {
134
- "content": " ",
135
- "lstrip": false,
136
- "normalized": false,
137
- "rstrip": false,
138
- "single_word": false,
139
- "special": false
140
- },
141
- "17": {
142
- "content": " ",
143
- "lstrip": false,
144
- "normalized": false,
145
- "rstrip": false,
146
- "single_word": false,
147
- "special": false
148
- },
149
- "18": {
150
- "content": " ",
151
- "lstrip": false,
152
- "normalized": false,
153
- "rstrip": false,
154
- "single_word": false,
155
- "special": false
156
- },
157
- "19": {
158
- "content": " ",
159
- "lstrip": false,
160
- "normalized": false,
161
- "rstrip": false,
162
- "single_word": false,
163
- "special": false
164
- },
165
- "20": {
166
- "content": " ",
167
- "lstrip": false,
168
- "normalized": false,
169
- "rstrip": false,
170
- "single_word": false,
171
- "special": false
172
- },
173
- "21": {
174
- "content": " ",
175
- "lstrip": false,
176
- "normalized": false,
177
- "rstrip": false,
178
- "single_word": false,
179
- "special": false
180
- },
181
- "22": {
182
- "content": " ",
183
- "lstrip": false,
184
- "normalized": false,
185
- "rstrip": false,
186
- "single_word": false,
187
- "special": false
188
- },
189
- "23": {
190
- "content": " ",
191
- "lstrip": false,
192
- "normalized": false,
193
- "rstrip": false,
194
- "single_word": false,
195
- "special": false
196
- },
197
- "24": {
198
- "content": " ",
199
- "lstrip": false,
200
- "normalized": false,
201
- "rstrip": false,
202
- "single_word": false,
203
- "special": false
204
- },
205
- "25": {
206
- "content": " ",
207
- "lstrip": false,
208
- "normalized": false,
209
- "rstrip": false,
210
- "single_word": false,
211
- "special": false
212
- },
213
- "26": {
214
- "content": " ",
215
- "lstrip": false,
216
- "normalized": false,
217
- "rstrip": false,
218
- "single_word": false,
219
- "special": false
220
- },
221
- "27": {
222
- "content": " ",
223
- "lstrip": false,
224
- "normalized": false,
225
- "rstrip": false,
226
- "single_word": false,
227
- "special": false
228
- },
229
- "28": {
230
- "content": " ",
231
- "lstrip": false,
232
- "normalized": false,
233
- "rstrip": false,
234
- "single_word": false,
235
- "special": false
236
- },
237
- "29": {
238
- "content": " ",
239
- "lstrip": false,
240
- "normalized": false,
241
- "rstrip": false,
242
- "single_word": false,
243
- "special": false
244
- },
245
- "30": {
246
- "content": " ",
247
- "lstrip": false,
248
- "normalized": false,
249
- "rstrip": false,
250
- "single_word": false,
251
- "special": false
252
- },
253
- "31": {
254
- "content": " ",
255
- "lstrip": false,
256
- "normalized": false,
257
- "rstrip": false,
258
- "single_word": false,
259
- "special": false
260
- },
261
- "32": {
262
- "content": " ",
263
- "lstrip": false,
264
- "normalized": false,
265
- "rstrip": false,
266
- "single_word": false,
267
- "special": false
268
- },
269
- "33": {
270
- "content": " ",
271
- "lstrip": false,
272
- "normalized": false,
273
- "rstrip": false,
274
- "single_word": false,
275
- "special": false
276
- },
277
- "34": {
278
- "content": "\t\t\t\t\t\t\t\t\t",
279
- "lstrip": false,
280
- "normalized": false,
281
- "rstrip": false,
282
- "single_word": false,
283
- "special": false
284
- },
285
- "35": {
286
- "content": "\t\t\t\t\t\t\t\t",
287
- "lstrip": false,
288
- "normalized": false,
289
- "rstrip": false,
290
- "single_word": false,
291
- "special": false
292
- },
293
- "36": {
294
- "content": "\t\t\t\t\t\t\t",
295
- "lstrip": false,
296
- "normalized": false,
297
- "rstrip": false,
298
- "single_word": false,
299
- "special": false
300
- },
301
- "37": {
302
- "content": "\t\t\t\t\t\t",
303
- "lstrip": false,
304
- "normalized": false,
305
- "rstrip": false,
306
- "single_word": false,
307
- "special": false
308
- },
309
- "38": {
310
- "content": "\t\t\t\t\t",
311
- "lstrip": false,
312
- "normalized": false,
313
- "rstrip": false,
314
- "single_word": false,
315
- "special": false
316
- },
317
- "39": {
318
- "content": "\t\t\t\t",
319
- "lstrip": false,
320
- "normalized": false,
321
- "rstrip": false,
322
- "single_word": false,
323
- "special": false
324
- },
325
- "40": {
326
- "content": "\t\t\t",
327
- "lstrip": false,
328
- "normalized": false,
329
- "rstrip": false,
330
- "single_word": false,
331
- "special": false
332
- },
333
- "41": {
334
- "content": "\t\t",
335
- "lstrip": false,
336
- "normalized": false,
337
- "rstrip": false,
338
- "single_word": false,
339
- "special": false
340
- },
341
- "42": {
342
- "content": "<|endoftext|>",
343
- "lstrip": false,
344
- "normalized": false,
345
- "rstrip": false,
346
- "single_word": false,
347
- "special": true
348
- },
349
- "43": {
350
- "content": "<|c|>",
351
- "lstrip": false,
352
- "normalized": false,
353
- "rstrip": false,
354
- "single_word": false,
355
- "special": true
356
- },
357
- "44": {
358
- "content": "<|c++|>",
359
- "lstrip": false,
360
- "normalized": false,
361
- "rstrip": false,
362
- "single_word": false,
363
- "special": true
364
- },
365
- "45": {
366
- "content": "<|python|>",
367
- "lstrip": false,
368
- "normalized": false,
369
- "rstrip": false,
370
- "single_word": false,
371
- "special": true
372
- },
373
- "46": {
374
- "content": "<|javascript|>",
375
- "lstrip": false,
376
- "normalized": false,
377
- "rstrip": false,
378
- "single_word": false,
379
- "special": true
380
- },
381
- "47": {
382
- "content": "<|markdown|>",
383
- "lstrip": false,
384
- "normalized": false,
385
- "rstrip": false,
386
- "single_word": false,
387
- "special": true
388
- },
389
- "48": {
390
- "content": "<|html|>",
391
- "lstrip": false,
392
- "normalized": false,
393
- "rstrip": false,
394
- "single_word": false,
395
- "special": true
396
- },
397
- "49": {
398
- "content": "<|css|>",
399
- "lstrip": false,
400
- "normalized": false,
401
- "rstrip": false,
402
- "single_word": false,
403
- "special": true
404
- },
405
- "50": {
406
- "content": "<|vue|>",
407
- "lstrip": false,
408
- "normalized": false,
409
- "rstrip": false,
410
- "single_word": false,
411
- "special": true
412
- },
413
- "51": {
414
- "content": "<|java|>",
415
- "lstrip": false,
416
- "normalized": false,
417
- "rstrip": false,
418
- "single_word": false,
419
- "special": true
420
- },
421
- "52": {
422
- "content": "PI:URL",
423
- "lstrip": false,
424
- "normalized": false,
425
- "rstrip": false,
426
- "single_word": false,
427
- "special": true
428
- },
429
- "53": {
430
- "content": "PI:EMAIL",
431
- "lstrip": false,
432
- "normalized": false,
433
- "rstrip": false,
434
- "single_word": false,
435
- "special": true
436
- },
437
- "54": {
438
- "content": "PI:ACCOUNT_NUM",
439
- "lstrip": false,
440
- "normalized": false,
441
- "rstrip": false,
442
- "single_word": false,
443
- "special": true
444
- },
445
- "55": {
446
- "content": "PI:PHONE_NUM",
447
- "lstrip": false,
448
- "normalized": false,
449
- "rstrip": false,
450
- "single_word": false,
451
- "special": true
452
- },
453
- "56": {
454
- "content": "PI:BUSINESS_NUM",
455
- "lstrip": false,
456
- "normalized": false,
457
- "rstrip": false,
458
- "single_word": false,
459
- "special": true
460
- },
461
- "57": {
462
- "content": "PI:ANNON",
463
- "lstrip": false,
464
- "normalized": false,
465
- "rstrip": false,
466
- "single_word": false,
467
- "special": true
468
- },
469
- "58": {
470
- "content": "PI:KEY",
471
- "lstrip": false,
472
- "normalized": false,
473
- "rstrip": false,
474
- "single_word": false,
475
- "special": true
476
- },
477
- "59": {
478
- "content": "PI:ID",
479
- "lstrip": false,
480
- "normalized": false,
481
- "rstrip": false,
482
- "single_word": false,
483
- "special": true
484
- },
485
- "60": {
486
- "content": "PI:IP_ADDRESS",
487
- "lstrip": false,
488
- "normalized": false,
489
- "rstrip": false,
490
- "single_word": false,
491
- "special": true
492
- },
493
- "61": {
494
- "content": "PI:USER",
495
- "lstrip": false,
496
- "normalized": false,
497
- "rstrip": false,
498
- "single_word": false,
499
- "special": true
500
- },
501
- "62": {
502
- "content": "[unused0]",
503
- "lstrip": false,
504
- "normalized": false,
505
- "rstrip": false,
506
- "single_word": false,
507
- "special": true
508
- },
509
- "63": {
510
- "content": "[unused1]",
511
- "lstrip": false,
512
- "normalized": false,
513
- "rstrip": false,
514
- "single_word": false,
515
- "special": true
516
- },
517
- "64": {
518
- "content": "[unused2]",
519
- "lstrip": false,
520
- "normalized": false,
521
- "rstrip": false,
522
- "single_word": false,
523
- "special": true
524
- },
525
- "65": {
526
- "content": "[unused3]",
527
- "lstrip": false,
528
- "normalized": false,
529
- "rstrip": false,
530
- "single_word": false,
531
- "special": true
532
- },
533
- "66": {
534
- "content": "[unused4]",
535
- "lstrip": false,
536
- "normalized": false,
537
- "rstrip": false,
538
- "single_word": false,
539
- "special": true
540
- },
541
- "67": {
542
- "content": "[unused5]",
543
- "lstrip": false,
544
- "normalized": false,
545
- "rstrip": false,
546
- "single_word": false,
547
- "special": true
548
- },
549
- "68": {
550
- "content": "[unused6]",
551
- "lstrip": false,
552
- "normalized": false,
553
- "rstrip": false,
554
- "single_word": false,
555
- "special": true
556
- },
557
- "69": {
558
- "content": "[unused7]",
559
- "lstrip": false,
560
- "normalized": false,
561
- "rstrip": false,
562
- "single_word": false,
563
- "special": true
564
- },
565
- "70": {
566
- "content": "[unused8]",
567
- "lstrip": false,
568
- "normalized": false,
569
- "rstrip": false,
570
- "single_word": false,
571
- "special": true
572
- },
573
- "71": {
574
- "content": "[unused9]",
575
- "lstrip": false,
576
- "normalized": false,
577
- "rstrip": false,
578
- "single_word": false,
579
- "special": true
580
- },
581
- "72": {
582
- "content": "[unused10]",
583
- "lstrip": false,
584
- "normalized": false,
585
- "rstrip": false,
586
- "single_word": false,
587
- "special": true
588
- },
589
- "73": {
590
- "content": "[unused11]",
591
- "lstrip": false,
592
- "normalized": false,
593
- "rstrip": false,
594
- "single_word": false,
595
- "special": true
596
- },
597
- "74": {
598
- "content": "[unused12]",
599
- "lstrip": false,
600
- "normalized": false,
601
- "rstrip": false,
602
- "single_word": false,
603
- "special": true
604
- },
605
- "75": {
606
- "content": "[unused13]",
607
- "lstrip": false,
608
- "normalized": false,
609
- "rstrip": false,
610
- "single_word": false,
611
- "special": true
612
- },
613
- "76": {
614
- "content": "[unused14]",
615
- "lstrip": false,
616
- "normalized": false,
617
- "rstrip": false,
618
- "single_word": false,
619
- "special": true
620
- },
621
- "77": {
622
- "content": "[unused15]",
623
- "lstrip": false,
624
- "normalized": false,
625
- "rstrip": false,
626
- "single_word": false,
627
- "special": true
628
- },
629
- "78": {
630
- "content": "[unused16]",
631
- "lstrip": false,
632
- "normalized": false,
633
- "rstrip": false,
634
- "single_word": false,
635
- "special": true
636
- },
637
- "79": {
638
- "content": "[unused17]",
639
- "lstrip": false,
640
- "normalized": false,
641
- "rstrip": false,
642
- "single_word": false,
643
- "special": true
644
- },
645
- "80": {
646
- "content": "[unused18]",
647
- "lstrip": false,
648
- "normalized": false,
649
- "rstrip": false,
650
- "single_word": false,
651
- "special": true
652
- },
653
- "81": {
654
- "content": "[unused19]",
655
- "lstrip": false,
656
- "normalized": false,
657
- "rstrip": false,
658
- "single_word": false,
659
- "special": true
660
- },
661
- "82": {
662
- "content": "[unused20]",
663
- "lstrip": false,
664
- "normalized": false,
665
- "rstrip": false,
666
- "single_word": false,
667
- "special": true
668
- },
669
- "83": {
670
- "content": "[unused21]",
671
- "lstrip": false,
672
- "normalized": false,
673
- "rstrip": false,
674
- "single_word": false,
675
- "special": true
676
- },
677
- "84": {
678
- "content": "[unused22]",
679
- "lstrip": false,
680
- "normalized": false,
681
- "rstrip": false,
682
- "single_word": false,
683
- "special": true
684
- },
685
- "85": {
686
- "content": "[unused23]",
687
- "lstrip": false,
688
- "normalized": false,
689
- "rstrip": false,
690
- "single_word": false,
691
- "special": true
692
- },
693
- "86": {
694
- "content": "[unused24]",
695
- "lstrip": false,
696
- "normalized": false,
697
- "rstrip": false,
698
- "single_word": false,
699
- "special": true
700
- },
701
- "87": {
702
- "content": "[unused25]",
703
- "lstrip": false,
704
- "normalized": false,
705
- "rstrip": false,
706
- "single_word": false,
707
- "special": true
708
- },
709
- "88": {
710
- "content": "[unused26]",
711
- "lstrip": false,
712
- "normalized": false,
713
- "rstrip": false,
714
- "single_word": false,
715
- "special": true
716
- },
717
- "89": {
718
- "content": "[unused27]",
719
- "lstrip": false,
720
- "normalized": false,
721
- "rstrip": false,
722
- "single_word": false,
723
- "special": true
724
- },
725
- "90": {
726
- "content": "[unused28]",
727
- "lstrip": false,
728
- "normalized": false,
729
- "rstrip": false,
730
- "single_word": false,
731
- "special": true
732
- },
733
- "91": {
734
- "content": "[unused29]",
735
- "lstrip": false,
736
- "normalized": false,
737
- "rstrip": false,
738
- "single_word": false,
739
- "special": true
740
- },
741
- "92": {
742
- "content": "[unused30]",
743
- "lstrip": false,
744
- "normalized": false,
745
- "rstrip": false,
746
- "single_word": false,
747
- "special": true
748
- },
749
- "93": {
750
- "content": "[unused31]",
751
- "lstrip": false,
752
- "normalized": false,
753
- "rstrip": false,
754
- "single_word": false,
755
- "special": true
756
- },
757
- "94": {
758
- "content": "[unused32]",
759
- "lstrip": false,
760
- "normalized": false,
761
- "rstrip": false,
762
- "single_word": false,
763
- "special": true
764
- },
765
- "95": {
766
- "content": "[unused33]",
767
- "lstrip": false,
768
- "normalized": false,
769
- "rstrip": false,
770
- "single_word": false,
771
- "special": true
772
- },
773
- "96": {
774
- "content": "[unused34]",
775
- "lstrip": false,
776
- "normalized": false,
777
- "rstrip": false,
778
- "single_word": false,
779
- "special": true
780
- },
781
- "97": {
782
- "content": "[unused35]",
783
- "lstrip": false,
784
- "normalized": false,
785
- "rstrip": false,
786
- "single_word": false,
787
- "special": true
788
- },
789
- "98": {
790
- "content": "[unused36]",
791
- "lstrip": false,
792
- "normalized": false,
793
- "rstrip": false,
794
- "single_word": false,
795
- "special": true
796
- },
797
- "99": {
798
- "content": "[unused37]",
799
- "lstrip": false,
800
- "normalized": false,
801
- "rstrip": false,
802
- "single_word": false,
803
- "special": true
804
- },
805
- "100": {
806
- "content": "[unused38]",
807
- "lstrip": false,
808
- "normalized": false,
809
- "rstrip": false,
810
- "single_word": false,
811
- "special": true
812
- },
813
- "101": {
814
- "content": "[unused39]",
815
- "lstrip": false,
816
- "normalized": false,
817
- "rstrip": false,
818
- "single_word": false,
819
- "special": true
820
- },
821
- "102": {
822
- "content": "[unused40]",
823
- "lstrip": false,
824
- "normalized": false,
825
- "rstrip": false,
826
- "single_word": false,
827
- "special": true
828
- },
829
- "103": {
830
- "content": "[unused41]",
831
- "lstrip": false,
832
- "normalized": false,
833
- "rstrip": false,
834
- "single_word": false,
835
- "special": true
836
- },
837
- "104": {
838
- "content": "[unused42]",
839
- "lstrip": false,
840
- "normalized": false,
841
- "rstrip": false,
842
- "single_word": false,
843
- "special": true
844
- },
845
- "105": {
846
- "content": "[unused43]",
847
- "lstrip": false,
848
- "normalized": false,
849
- "rstrip": false,
850
- "single_word": false,
851
- "special": true
852
- },
853
- "106": {
854
- "content": "[unused44]",
855
- "lstrip": false,
856
- "normalized": false,
857
- "rstrip": false,
858
- "single_word": false,
859
- "special": true
860
- },
861
- "107": {
862
- "content": "[unused45]",
863
- "lstrip": false,
864
- "normalized": false,
865
- "rstrip": false,
866
- "single_word": false,
867
- "special": true
868
- },
869
- "108": {
870
- "content": "[unused46]",
871
- "lstrip": false,
872
- "normalized": false,
873
- "rstrip": false,
874
- "single_word": false,
875
- "special": true
876
- },
877
- "109": {
878
- "content": "[unused47]",
879
- "lstrip": false,
880
- "normalized": false,
881
- "rstrip": false,
882
- "single_word": false,
883
- "special": true
884
- },
885
- "110": {
886
- "content": "[unused48]",
887
- "lstrip": false,
888
- "normalized": false,
889
- "rstrip": false,
890
- "single_word": false,
891
- "special": true
892
- },
893
- "111": {
894
- "content": "[unused49]",
895
- "lstrip": false,
896
- "normalized": false,
897
- "rstrip": false,
898
- "single_word": false,
899
- "special": true
900
- },
901
- "112": {
902
- "content": "[unused50]",
903
- "lstrip": false,
904
- "normalized": false,
905
- "rstrip": false,
906
- "single_word": false,
907
- "special": true
908
- },
909
- "113": {
910
- "content": "[unused51]",
911
- "lstrip": false,
912
- "normalized": false,
913
- "rstrip": false,
914
- "single_word": false,
915
- "special": true
916
- },
917
- "114": {
918
- "content": "[unused52]",
919
- "lstrip": false,
920
- "normalized": false,
921
- "rstrip": false,
922
- "single_word": false,
923
- "special": true
924
- },
925
- "115": {
926
- "content": "[unused53]",
927
- "lstrip": false,
928
- "normalized": false,
929
- "rstrip": false,
930
- "single_word": false,
931
- "special": true
932
- },
933
- "116": {
934
- "content": "[unused54]",
935
- "lstrip": false,
936
- "normalized": false,
937
- "rstrip": false,
938
- "single_word": false,
939
- "special": true
940
- },
941
- "117": {
942
- "content": "[unused55]",
943
- "lstrip": false,
944
- "normalized": false,
945
- "rstrip": false,
946
- "single_word": false,
947
- "special": true
948
- },
949
- "118": {
950
- "content": "[unused56]",
951
- "lstrip": false,
952
- "normalized": false,
953
- "rstrip": false,
954
- "single_word": false,
955
- "special": true
956
- },
957
- "119": {
958
- "content": "[unused57]",
959
- "lstrip": false,
960
- "normalized": false,
961
- "rstrip": false,
962
- "single_word": false,
963
- "special": true
964
- },
965
- "120": {
966
- "content": "[unused58]",
967
- "lstrip": false,
968
- "normalized": false,
969
- "rstrip": false,
970
- "single_word": false,
971
- "special": true
972
- },
973
- "121": {
974
- "content": "[unused59]",
975
- "lstrip": false,
976
- "normalized": false,
977
- "rstrip": false,
978
- "single_word": false,
979
- "special": true
980
- },
981
- "122": {
982
- "content": "[unused60]",
983
- "lstrip": false,
984
- "normalized": false,
985
- "rstrip": false,
986
- "single_word": false,
987
- "special": true
988
- },
989
- "123": {
990
- "content": "[unused61]",
991
- "lstrip": false,
992
- "normalized": false,
993
- "rstrip": false,
994
- "single_word": false,
995
- "special": true
996
- },
997
- "124": {
998
- "content": "[unused62]",
999
- "lstrip": false,
1000
- "normalized": false,
1001
- "rstrip": false,
1002
- "single_word": false,
1003
- "special": true
1004
- },
1005
- "125": {
1006
- "content": "[unused63]",
1007
- "lstrip": false,
1008
- "normalized": false,
1009
- "rstrip": false,
1010
- "single_word": false,
1011
- "special": true
1012
- },
1013
- "126": {
1014
- "content": "[unused64]",
1015
- "lstrip": false,
1016
- "normalized": false,
1017
- "rstrip": false,
1018
- "single_word": false,
1019
- "special": true
1020
- },
1021
- "127": {
1022
- "content": "[unused65]",
1023
- "lstrip": false,
1024
- "normalized": false,
1025
- "rstrip": false,
1026
- "single_word": false,
1027
- "special": true
1028
- },
1029
- "128": {
1030
- "content": "[unused66]",
1031
- "lstrip": false,
1032
- "normalized": false,
1033
- "rstrip": false,
1034
- "single_word": false,
1035
- "special": true
1036
- },
1037
- "129": {
1038
- "content": "[unused67]",
1039
- "lstrip": false,
1040
- "normalized": false,
1041
- "rstrip": false,
1042
- "single_word": false,
1043
- "special": true
1044
- },
1045
- "130": {
1046
- "content": "[unused68]",
1047
- "lstrip": false,
1048
- "normalized": false,
1049
- "rstrip": false,
1050
- "single_word": false,
1051
- "special": true
1052
- },
1053
- "131": {
1054
- "content": "[unused69]",
1055
- "lstrip": false,
1056
- "normalized": false,
1057
- "rstrip": false,
1058
- "single_word": false,
1059
- "special": true
1060
- },
1061
- "132": {
1062
- "content": "[unused70]",
1063
- "lstrip": false,
1064
- "normalized": false,
1065
- "rstrip": false,
1066
- "single_word": false,
1067
- "special": true
1068
- },
1069
- "133": {
1070
- "content": "[unused71]",
1071
- "lstrip": false,
1072
- "normalized": false,
1073
- "rstrip": false,
1074
- "single_word": false,
1075
- "special": true
1076
- },
1077
- "134": {
1078
- "content": "[unused72]",
1079
- "lstrip": false,
1080
- "normalized": false,
1081
- "rstrip": false,
1082
- "single_word": false,
1083
- "special": true
1084
- },
1085
- "135": {
1086
- "content": "[unused73]",
1087
- "lstrip": false,
1088
- "normalized": false,
1089
- "rstrip": false,
1090
- "single_word": false,
1091
- "special": true
1092
- },
1093
- "136": {
1094
- "content": "[unused74]",
1095
- "lstrip": false,
1096
- "normalized": false,
1097
- "rstrip": false,
1098
- "single_word": false,
1099
- "special": true
1100
- },
1101
- "137": {
1102
- "content": "[unused75]",
1103
- "lstrip": false,
1104
- "normalized": false,
1105
- "rstrip": false,
1106
- "single_word": false,
1107
- "special": true
1108
- },
1109
- "138": {
1110
- "content": "[unused76]",
1111
- "lstrip": false,
1112
- "normalized": false,
1113
- "rstrip": false,
1114
- "single_word": false,
1115
- "special": true
1116
- },
1117
- "139": {
1118
- "content": "[unused77]",
1119
- "lstrip": false,
1120
- "normalized": false,
1121
- "rstrip": false,
1122
- "single_word": false,
1123
- "special": true
1124
- },
1125
- "140": {
1126
- "content": "[unused78]",
1127
- "lstrip": false,
1128
- "normalized": false,
1129
- "rstrip": false,
1130
- "single_word": false,
1131
- "special": true
1132
- },
1133
- "141": {
1134
- "content": "[unused79]",
1135
- "lstrip": false,
1136
- "normalized": false,
1137
- "rstrip": false,
1138
- "single_word": false,
1139
- "special": true
1140
- },
1141
- "142": {
1142
- "content": "[unused80]",
1143
- "lstrip": false,
1144
- "normalized": false,
1145
- "rstrip": false,
1146
- "single_word": false,
1147
- "special": true
1148
- },
1149
- "143": {
1150
- "content": "[unused81]",
1151
- "lstrip": false,
1152
- "normalized": false,
1153
- "rstrip": false,
1154
- "single_word": false,
1155
- "special": true
1156
- },
1157
- "144": {
1158
- "content": "[unused82]",
1159
- "lstrip": false,
1160
- "normalized": false,
1161
- "rstrip": false,
1162
- "single_word": false,
1163
- "special": true
1164
- },
1165
- "145": {
1166
- "content": "[unused83]",
1167
- "lstrip": false,
1168
- "normalized": false,
1169
- "rstrip": false,
1170
- "single_word": false,
1171
- "special": true
1172
- },
1173
- "146": {
1174
- "content": "[unused84]",
1175
- "lstrip": false,
1176
- "normalized": false,
1177
- "rstrip": false,
1178
- "single_word": false,
1179
- "special": true
1180
- },
1181
- "147": {
1182
- "content": "[unused85]",
1183
- "lstrip": false,
1184
- "normalized": false,
1185
- "rstrip": false,
1186
- "single_word": false,
1187
- "special": true
1188
- },
1189
- "148": {
1190
- "content": "[unused86]",
1191
- "lstrip": false,
1192
- "normalized": false,
1193
- "rstrip": false,
1194
- "single_word": false,
1195
- "special": true
1196
- },
1197
- "149": {
1198
- "content": "[unused87]",
1199
- "lstrip": false,
1200
- "normalized": false,
1201
- "rstrip": false,
1202
- "single_word": false,
1203
- "special": true
1204
- },
1205
- "150": {
1206
- "content": "[unused88]",
1207
- "lstrip": false,
1208
- "normalized": false,
1209
- "rstrip": false,
1210
- "single_word": false,
1211
- "special": true
1212
- },
1213
- "151": {
1214
- "content": "[unused89]",
1215
- "lstrip": false,
1216
- "normalized": false,
1217
- "rstrip": false,
1218
- "single_word": false,
1219
- "special": true
1220
- },
1221
- "152": {
1222
- "content": "[unused90]",
1223
- "lstrip": false,
1224
- "normalized": false,
1225
- "rstrip": false,
1226
- "single_word": false,
1227
- "special": true
1228
- },
1229
- "153": {
1230
- "content": "[unused91]",
1231
- "lstrip": false,
1232
- "normalized": false,
1233
- "rstrip": false,
1234
- "single_word": false,
1235
- "special": true
1236
- },
1237
- "154": {
1238
- "content": "[unused92]",
1239
- "lstrip": false,
1240
- "normalized": false,
1241
- "rstrip": false,
1242
- "single_word": false,
1243
- "special": true
1244
- },
1245
- "155": {
1246
- "content": "[unused93]",
1247
- "lstrip": false,
1248
- "normalized": false,
1249
- "rstrip": false,
1250
- "single_word": false,
1251
- "special": true
1252
- },
1253
- "156": {
1254
- "content": "[unused94]",
1255
- "lstrip": false,
1256
- "normalized": false,
1257
- "rstrip": false,
1258
- "single_word": false,
1259
- "special": true
1260
- },
1261
- "157": {
1262
- "content": "[unused95]",
1263
- "lstrip": false,
1264
- "normalized": false,
1265
- "rstrip": false,
1266
- "single_word": false,
1267
- "special": true
1268
- },
1269
- "158": {
1270
- "content": "[unused96]",
1271
- "lstrip": false,
1272
- "normalized": false,
1273
- "rstrip": false,
1274
- "single_word": false,
1275
- "special": true
1276
- },
1277
- "159": {
1278
- "content": "[unused97]",
1279
- "lstrip": false,
1280
- "normalized": false,
1281
- "rstrip": false,
1282
- "single_word": false,
1283
- "special": true
1284
- },
1285
- "160": {
1286
- "content": "[unused98]",
1287
- "lstrip": false,
1288
- "normalized": false,
1289
- "rstrip": false,
1290
- "single_word": false,
1291
- "special": true
1292
- },
1293
- "161": {
1294
- "content": "[unused99]",
1295
- "lstrip": false,
1296
- "normalized": false,
1297
- "rstrip": false,
1298
- "single_word": false,
1299
- "special": true
1300
- },
1301
- "162": {
1302
- "content": "[extra_id_0]",
1303
- "lstrip": false,
1304
- "normalized": false,
1305
- "rstrip": false,
1306
- "single_word": false,
1307
- "special": true
1308
- },
1309
- "163": {
1310
- "content": "[extra_id_1]",
1311
- "lstrip": false,
1312
- "normalized": false,
1313
- "rstrip": false,
1314
- "single_word": false,
1315
- "special": true
1316
- },
1317
- "164": {
1318
- "content": "[extra_id_2]",
1319
- "lstrip": false,
1320
- "normalized": false,
1321
- "rstrip": false,
1322
- "single_word": false,
1323
- "special": true
1324
- },
1325
- "165": {
1326
- "content": "[extra_id_3]",
1327
- "lstrip": false,
1328
- "normalized": false,
1329
- "rstrip": false,
1330
- "single_word": false,
1331
- "special": true
1332
- },
1333
- "166": {
1334
- "content": "[extra_id_4]",
1335
- "lstrip": false,
1336
- "normalized": false,
1337
- "rstrip": false,
1338
- "single_word": false,
1339
- "special": true
1340
- },
1341
- "167": {
1342
- "content": "[extra_id_5]",
1343
- "lstrip": false,
1344
- "normalized": false,
1345
- "rstrip": false,
1346
- "single_word": false,
1347
- "special": true
1348
- },
1349
- "168": {
1350
- "content": "[extra_id_6]",
1351
- "lstrip": false,
1352
- "normalized": false,
1353
- "rstrip": false,
1354
- "single_word": false,
1355
- "special": true
1356
- },
1357
- "169": {
1358
- "content": "[extra_id_7]",
1359
- "lstrip": false,
1360
- "normalized": false,
1361
- "rstrip": false,
1362
- "single_word": false,
1363
- "special": true
1364
- },
1365
- "170": {
1366
- "content": "[extra_id_8]",
1367
- "lstrip": false,
1368
- "normalized": false,
1369
- "rstrip": false,
1370
- "single_word": false,
1371
- "special": true
1372
- },
1373
- "171": {
1374
- "content": "[extra_id_9]",
1375
- "lstrip": false,
1376
- "normalized": false,
1377
- "rstrip": false,
1378
- "single_word": false,
1379
- "special": true
1380
- },
1381
- "172": {
1382
- "content": "[extra_id_10]",
1383
- "lstrip": false,
1384
- "normalized": false,
1385
- "rstrip": false,
1386
- "single_word": false,
1387
- "special": true
1388
- },
1389
- "173": {
1390
- "content": "[extra_id_11]",
1391
- "lstrip": false,
1392
- "normalized": false,
1393
- "rstrip": false,
1394
- "single_word": false,
1395
- "special": true
1396
- },
1397
- "174": {
1398
- "content": "[extra_id_12]",
1399
- "lstrip": false,
1400
- "normalized": false,
1401
- "rstrip": false,
1402
- "single_word": false,
1403
- "special": true
1404
- },
1405
- "175": {
1406
- "content": "[extra_id_13]",
1407
- "lstrip": false,
1408
- "normalized": false,
1409
- "rstrip": false,
1410
- "single_word": false,
1411
- "special": true
1412
- },
1413
- "176": {
1414
- "content": "[extra_id_14]",
1415
- "lstrip": false,
1416
- "normalized": false,
1417
- "rstrip": false,
1418
- "single_word": false,
1419
- "special": true
1420
- },
1421
- "177": {
1422
- "content": "[extra_id_15]",
1423
- "lstrip": false,
1424
- "normalized": false,
1425
- "rstrip": false,
1426
- "single_word": false,
1427
- "special": true
1428
- },
1429
- "178": {
1430
- "content": "[extra_id_16]",
1431
- "lstrip": false,
1432
- "normalized": false,
1433
- "rstrip": false,
1434
- "single_word": false,
1435
- "special": true
1436
- },
1437
- "179": {
1438
- "content": "[extra_id_17]",
1439
- "lstrip": false,
1440
- "normalized": false,
1441
- "rstrip": false,
1442
- "single_word": false,
1443
- "special": true
1444
- },
1445
- "180": {
1446
- "content": "[extra_id_18]",
1447
- "lstrip": false,
1448
- "normalized": false,
1449
- "rstrip": false,
1450
- "single_word": false,
1451
- "special": true
1452
- },
1453
- "181": {
1454
- "content": "[extra_id_19]",
1455
- "lstrip": false,
1456
- "normalized": false,
1457
- "rstrip": false,
1458
- "single_word": false,
1459
- "special": true
1460
- },
1461
- "182": {
1462
- "content": "[extra_id_20]",
1463
- "lstrip": false,
1464
- "normalized": false,
1465
- "rstrip": false,
1466
- "single_word": false,
1467
- "special": true
1468
- },
1469
- "183": {
1470
- "content": "[extra_id_21]",
1471
- "lstrip": false,
1472
- "normalized": false,
1473
- "rstrip": false,
1474
- "single_word": false,
1475
- "special": true
1476
- },
1477
- "184": {
1478
- "content": "[extra_id_22]",
1479
- "lstrip": false,
1480
- "normalized": false,
1481
- "rstrip": false,
1482
- "single_word": false,
1483
- "special": true
1484
- },
1485
- "185": {
1486
- "content": "[extra_id_23]",
1487
- "lstrip": false,
1488
- "normalized": false,
1489
- "rstrip": false,
1490
- "single_word": false,
1491
- "special": true
1492
- },
1493
- "186": {
1494
- "content": "[extra_id_24]",
1495
- "lstrip": false,
1496
- "normalized": false,
1497
- "rstrip": false,
1498
- "single_word": false,
1499
- "special": true
1500
- },
1501
- "187": {
1502
- "content": "[extra_id_25]",
1503
- "lstrip": false,
1504
- "normalized": false,
1505
- "rstrip": false,
1506
- "single_word": false,
1507
- "special": true
1508
- },
1509
- "188": {
1510
- "content": "[extra_id_26]",
1511
- "lstrip": false,
1512
- "normalized": false,
1513
- "rstrip": false,
1514
- "single_word": false,
1515
- "special": true
1516
- },
1517
- "189": {
1518
- "content": "[extra_id_27]",
1519
- "lstrip": false,
1520
- "normalized": false,
1521
- "rstrip": false,
1522
- "single_word": false,
1523
- "special": true
1524
- },
1525
- "190": {
1526
- "content": "[extra_id_28]",
1527
- "lstrip": false,
1528
- "normalized": false,
1529
- "rstrip": false,
1530
- "single_word": false,
1531
- "special": true
1532
- },
1533
- "191": {
1534
- "content": "[extra_id_29]",
1535
- "lstrip": false,
1536
- "normalized": false,
1537
- "rstrip": false,
1538
- "single_word": false,
1539
- "special": true
1540
- },
1541
- "192": {
1542
- "content": "[extra_id_30]",
1543
- "lstrip": false,
1544
- "normalized": false,
1545
- "rstrip": false,
1546
- "single_word": false,
1547
- "special": true
1548
- },
1549
- "193": {
1550
- "content": "[extra_id_31]",
1551
- "lstrip": false,
1552
- "normalized": false,
1553
- "rstrip": false,
1554
- "single_word": false,
1555
- "special": true
1556
- },
1557
- "194": {
1558
- "content": "[extra_id_32]",
1559
- "lstrip": false,
1560
- "normalized": false,
1561
- "rstrip": false,
1562
- "single_word": false,
1563
- "special": true
1564
- },
1565
- "195": {
1566
- "content": "[extra_id_33]",
1567
- "lstrip": false,
1568
- "normalized": false,
1569
- "rstrip": false,
1570
- "single_word": false,
1571
- "special": true
1572
- },
1573
- "196": {
1574
- "content": "[extra_id_34]",
1575
- "lstrip": false,
1576
- "normalized": false,
1577
- "rstrip": false,
1578
- "single_word": false,
1579
- "special": true
1580
- },
1581
- "197": {
1582
- "content": "[extra_id_35]",
1583
- "lstrip": false,
1584
- "normalized": false,
1585
- "rstrip": false,
1586
- "single_word": false,
1587
- "special": true
1588
- },
1589
- "198": {
1590
- "content": "[extra_id_36]",
1591
- "lstrip": false,
1592
- "normalized": false,
1593
- "rstrip": false,
1594
- "single_word": false,
1595
- "special": true
1596
- },
1597
- "199": {
1598
- "content": "[extra_id_37]",
1599
- "lstrip": false,
1600
- "normalized": false,
1601
- "rstrip": false,
1602
- "single_word": false,
1603
- "special": true
1604
- },
1605
- "200": {
1606
- "content": "[extra_id_38]",
1607
- "lstrip": false,
1608
- "normalized": false,
1609
- "rstrip": false,
1610
- "single_word": false,
1611
- "special": true
1612
- },
1613
- "201": {
1614
- "content": "[extra_id_39]",
1615
- "lstrip": false,
1616
- "normalized": false,
1617
- "rstrip": false,
1618
- "single_word": false,
1619
- "special": true
1620
- },
1621
- "202": {
1622
- "content": "[extra_id_40]",
1623
- "lstrip": false,
1624
- "normalized": false,
1625
- "rstrip": false,
1626
- "single_word": false,
1627
- "special": true
1628
- },
1629
- "203": {
1630
- "content": "[extra_id_41]",
1631
- "lstrip": false,
1632
- "normalized": false,
1633
- "rstrip": false,
1634
- "single_word": false,
1635
- "special": true
1636
- },
1637
- "204": {
1638
- "content": "[extra_id_42]",
1639
- "lstrip": false,
1640
- "normalized": false,
1641
- "rstrip": false,
1642
- "single_word": false,
1643
- "special": true
1644
- },
1645
- "205": {
1646
- "content": "[extra_id_43]",
1647
- "lstrip": false,
1648
- "normalized": false,
1649
- "rstrip": false,
1650
- "single_word": false,
1651
- "special": true
1652
- },
1653
- "206": {
1654
- "content": "[extra_id_44]",
1655
- "lstrip": false,
1656
- "normalized": false,
1657
- "rstrip": false,
1658
- "single_word": false,
1659
- "special": true
1660
- },
1661
- "207": {
1662
- "content": "[extra_id_45]",
1663
- "lstrip": false,
1664
- "normalized": false,
1665
- "rstrip": false,
1666
- "single_word": false,
1667
- "special": true
1668
- },
1669
- "208": {
1670
- "content": "[extra_id_46]",
1671
- "lstrip": false,
1672
- "normalized": false,
1673
- "rstrip": false,
1674
- "single_word": false,
1675
- "special": true
1676
- },
1677
- "209": {
1678
- "content": "[extra_id_47]",
1679
- "lstrip": false,
1680
- "normalized": false,
1681
- "rstrip": false,
1682
- "single_word": false,
1683
- "special": true
1684
- },
1685
- "210": {
1686
- "content": "[extra_id_48]",
1687
- "lstrip": false,
1688
- "normalized": false,
1689
- "rstrip": false,
1690
- "single_word": false,
1691
- "special": true
1692
- },
1693
- "211": {
1694
- "content": "[extra_id_49]",
1695
- "lstrip": false,
1696
- "normalized": false,
1697
- "rstrip": false,
1698
- "single_word": false,
1699
- "special": true
1700
- },
1701
- "212": {
1702
- "content": "[extra_id_50]",
1703
- "lstrip": false,
1704
- "normalized": false,
1705
- "rstrip": false,
1706
- "single_word": false,
1707
- "special": true
1708
- },
1709
- "213": {
1710
- "content": "[extra_id_51]",
1711
- "lstrip": false,
1712
- "normalized": false,
1713
- "rstrip": false,
1714
- "single_word": false,
1715
- "special": true
1716
- },
1717
- "214": {
1718
- "content": "[extra_id_52]",
1719
- "lstrip": false,
1720
- "normalized": false,
1721
- "rstrip": false,
1722
- "single_word": false,
1723
- "special": true
1724
- },
1725
- "215": {
1726
- "content": "[extra_id_53]",
1727
- "lstrip": false,
1728
- "normalized": false,
1729
- "rstrip": false,
1730
- "single_word": false,
1731
- "special": true
1732
- },
1733
- "216": {
1734
- "content": "[extra_id_54]",
1735
- "lstrip": false,
1736
- "normalized": false,
1737
- "rstrip": false,
1738
- "single_word": false,
1739
- "special": true
1740
- },
1741
- "217": {
1742
- "content": "[extra_id_55]",
1743
- "lstrip": false,
1744
- "normalized": false,
1745
- "rstrip": false,
1746
- "single_word": false,
1747
- "special": true
1748
- },
1749
- "218": {
1750
- "content": "[extra_id_56]",
1751
- "lstrip": false,
1752
- "normalized": false,
1753
- "rstrip": false,
1754
- "single_word": false,
1755
- "special": true
1756
- },
1757
- "219": {
1758
- "content": "[extra_id_57]",
1759
- "lstrip": false,
1760
- "normalized": false,
1761
- "rstrip": false,
1762
- "single_word": false,
1763
- "special": true
1764
- },
1765
- "220": {
1766
- "content": "[extra_id_58]",
1767
- "lstrip": false,
1768
- "normalized": false,
1769
- "rstrip": false,
1770
- "single_word": false,
1771
- "special": true
1772
- },
1773
- "221": {
1774
- "content": "[extra_id_59]",
1775
- "lstrip": false,
1776
- "normalized": false,
1777
- "rstrip": false,
1778
- "single_word": false,
1779
- "special": true
1780
- },
1781
- "222": {
1782
- "content": "[extra_id_60]",
1783
- "lstrip": false,
1784
- "normalized": false,
1785
- "rstrip": false,
1786
- "single_word": false,
1787
- "special": true
1788
- },
1789
- "223": {
1790
- "content": "[extra_id_61]",
1791
- "lstrip": false,
1792
- "normalized": false,
1793
- "rstrip": false,
1794
- "single_word": false,
1795
- "special": true
1796
- },
1797
- "224": {
1798
- "content": "[extra_id_62]",
1799
- "lstrip": false,
1800
- "normalized": false,
1801
- "rstrip": false,
1802
- "single_word": false,
1803
- "special": true
1804
- },
1805
- "225": {
1806
- "content": "[extra_id_63]",
1807
- "lstrip": false,
1808
- "normalized": false,
1809
- "rstrip": false,
1810
- "single_word": false,
1811
- "special": true
1812
- },
1813
- "226": {
1814
- "content": "[extra_id_64]",
1815
- "lstrip": false,
1816
- "normalized": false,
1817
- "rstrip": false,
1818
- "single_word": false,
1819
- "special": true
1820
- },
1821
- "227": {
1822
- "content": "[extra_id_65]",
1823
- "lstrip": false,
1824
- "normalized": false,
1825
- "rstrip": false,
1826
- "single_word": false,
1827
- "special": true
1828
- },
1829
- "228": {
1830
- "content": "[extra_id_66]",
1831
- "lstrip": false,
1832
- "normalized": false,
1833
- "rstrip": false,
1834
- "single_word": false,
1835
- "special": true
1836
- },
1837
- "229": {
1838
- "content": "[extra_id_67]",
1839
- "lstrip": false,
1840
- "normalized": false,
1841
- "rstrip": false,
1842
- "single_word": false,
1843
- "special": true
1844
- },
1845
- "230": {
1846
- "content": "[extra_id_68]",
1847
- "lstrip": false,
1848
- "normalized": false,
1849
- "rstrip": false,
1850
- "single_word": false,
1851
- "special": true
1852
- },
1853
- "231": {
1854
- "content": "[extra_id_69]",
1855
- "lstrip": false,
1856
- "normalized": false,
1857
- "rstrip": false,
1858
- "single_word": false,
1859
- "special": true
1860
- },
1861
- "232": {
1862
- "content": "[extra_id_70]",
1863
- "lstrip": false,
1864
- "normalized": false,
1865
- "rstrip": false,
1866
- "single_word": false,
1867
- "special": true
1868
- },
1869
- "233": {
1870
- "content": "[extra_id_71]",
1871
- "lstrip": false,
1872
- "normalized": false,
1873
- "rstrip": false,
1874
- "single_word": false,
1875
- "special": true
1876
- },
1877
- "234": {
1878
- "content": "[extra_id_72]",
1879
- "lstrip": false,
1880
- "normalized": false,
1881
- "rstrip": false,
1882
- "single_word": false,
1883
- "special": true
1884
- },
1885
- "235": {
1886
- "content": "[extra_id_73]",
1887
- "lstrip": false,
1888
- "normalized": false,
1889
- "rstrip": false,
1890
- "single_word": false,
1891
- "special": true
1892
- },
1893
- "236": {
1894
- "content": "[extra_id_74]",
1895
- "lstrip": false,
1896
- "normalized": false,
1897
- "rstrip": false,
1898
- "single_word": false,
1899
- "special": true
1900
- },
1901
- "237": {
1902
- "content": "[extra_id_75]",
1903
- "lstrip": false,
1904
- "normalized": false,
1905
- "rstrip": false,
1906
- "single_word": false,
1907
- "special": true
1908
- },
1909
- "238": {
1910
- "content": "[extra_id_76]",
1911
- "lstrip": false,
1912
- "normalized": false,
1913
- "rstrip": false,
1914
- "single_word": false,
1915
- "special": true
1916
- },
1917
- "239": {
1918
- "content": "[extra_id_77]",
1919
- "lstrip": false,
1920
- "normalized": false,
1921
- "rstrip": false,
1922
- "single_word": false,
1923
- "special": true
1924
- },
1925
- "240": {
1926
- "content": "[extra_id_78]",
1927
- "lstrip": false,
1928
- "normalized": false,
1929
- "rstrip": false,
1930
- "single_word": false,
1931
- "special": true
1932
- },
1933
- "241": {
1934
- "content": "[extra_id_79]",
1935
- "lstrip": false,
1936
- "normalized": false,
1937
- "rstrip": false,
1938
- "single_word": false,
1939
- "special": true
1940
- },
1941
- "242": {
1942
- "content": "[extra_id_80]",
1943
- "lstrip": false,
1944
- "normalized": false,
1945
- "rstrip": false,
1946
- "single_word": false,
1947
- "special": true
1948
- },
1949
- "243": {
1950
- "content": "[extra_id_81]",
1951
- "lstrip": false,
1952
- "normalized": false,
1953
- "rstrip": false,
1954
- "single_word": false,
1955
- "special": true
1956
- },
1957
- "244": {
1958
- "content": "[extra_id_82]",
1959
- "lstrip": false,
1960
- "normalized": false,
1961
- "rstrip": false,
1962
- "single_word": false,
1963
- "special": true
1964
- },
1965
- "245": {
1966
- "content": "[extra_id_83]",
1967
- "lstrip": false,
1968
- "normalized": false,
1969
- "rstrip": false,
1970
- "single_word": false,
1971
- "special": true
1972
- },
1973
- "246": {
1974
- "content": "[extra_id_84]",
1975
- "lstrip": false,
1976
- "normalized": false,
1977
- "rstrip": false,
1978
- "single_word": false,
1979
- "special": true
1980
- },
1981
- "247": {
1982
- "content": "[extra_id_85]",
1983
- "lstrip": false,
1984
- "normalized": false,
1985
- "rstrip": false,
1986
- "single_word": false,
1987
- "special": true
1988
- },
1989
- "248": {
1990
- "content": "[extra_id_86]",
1991
- "lstrip": false,
1992
- "normalized": false,
1993
- "rstrip": false,
1994
- "single_word": false,
1995
- "special": true
1996
- },
1997
- "249": {
1998
- "content": "[extra_id_87]",
1999
- "lstrip": false,
2000
- "normalized": false,
2001
- "rstrip": false,
2002
- "single_word": false,
2003
- "special": true
2004
- },
2005
- "250": {
2006
- "content": "[extra_id_88]",
2007
- "lstrip": false,
2008
- "normalized": false,
2009
- "rstrip": false,
2010
- "single_word": false,
2011
- "special": true
2012
- },
2013
- "251": {
2014
- "content": "[extra_id_89]",
2015
- "lstrip": false,
2016
- "normalized": false,
2017
- "rstrip": false,
2018
- "single_word": false,
2019
- "special": true
2020
- },
2021
- "252": {
2022
- "content": "[extra_id_90]",
2023
- "lstrip": false,
2024
- "normalized": false,
2025
- "rstrip": false,
2026
- "single_word": false,
2027
- "special": true
2028
- },
2029
- "253": {
2030
- "content": "[extra_id_91]",
2031
- "lstrip": false,
2032
- "normalized": false,
2033
- "rstrip": false,
2034
- "single_word": false,
2035
- "special": true
2036
- },
2037
- "254": {
2038
- "content": "[extra_id_92]",
2039
- "lstrip": false,
2040
- "normalized": false,
2041
- "rstrip": false,
2042
- "single_word": false,
2043
- "special": true
2044
- },
2045
- "255": {
2046
- "content": "[extra_id_93]",
2047
- "lstrip": false,
2048
- "normalized": false,
2049
- "rstrip": false,
2050
- "single_word": false,
2051
- "special": true
2052
- },
2053
- "256": {
2054
- "content": "[extra_id_94]",
2055
- "lstrip": false,
2056
- "normalized": false,
2057
- "rstrip": false,
2058
- "single_word": false,
2059
- "special": true
2060
- },
2061
- "257": {
2062
- "content": "[extra_id_95]",
2063
- "lstrip": false,
2064
- "normalized": false,
2065
- "rstrip": false,
2066
- "single_word": false,
2067
- "special": true
2068
- },
2069
- "258": {
2070
- "content": "[extra_id_96]",
2071
- "lstrip": false,
2072
- "normalized": false,
2073
- "rstrip": false,
2074
- "single_word": false,
2075
- "special": true
2076
- },
2077
- "259": {
2078
- "content": "[extra_id_97]",
2079
- "lstrip": false,
2080
- "normalized": false,
2081
- "rstrip": false,
2082
- "single_word": false,
2083
- "special": true
2084
- },
2085
- "260": {
2086
- "content": "[extra_id_98]",
2087
- "lstrip": false,
2088
- "normalized": false,
2089
- "rstrip": false,
2090
- "single_word": false,
2091
- "special": true
2092
- },
2093
- "261": {
2094
- "content": "[extra_id_99]",
2095
- "lstrip": false,
2096
- "normalized": false,
2097
- "rstrip": false,
2098
- "single_word": false,
2099
- "special": true
2100
- },
2101
- "262": {
2102
- "content": "[extra_id_100]",
2103
- "lstrip": false,
2104
- "normalized": false,
2105
- "rstrip": false,
2106
- "single_word": false,
2107
- "special": true
2108
- },
2109
- "263": {
2110
- "content": "[extra_id_101]",
2111
- "lstrip": false,
2112
- "normalized": false,
2113
- "rstrip": false,
2114
- "single_word": false,
2115
- "special": true
2116
- },
2117
- "264": {
2118
- "content": "[extra_id_102]",
2119
- "lstrip": false,
2120
- "normalized": false,
2121
- "rstrip": false,
2122
- "single_word": false,
2123
- "special": true
2124
- },
2125
- "265": {
2126
- "content": "[extra_id_103]",
2127
- "lstrip": false,
2128
- "normalized": false,
2129
- "rstrip": false,
2130
- "single_word": false,
2131
- "special": true
2132
- },
2133
- "266": {
2134
- "content": "[extra_id_104]",
2135
- "lstrip": false,
2136
- "normalized": false,
2137
- "rstrip": false,
2138
- "single_word": false,
2139
- "special": true
2140
- },
2141
- "267": {
2142
- "content": "[extra_id_105]",
2143
- "lstrip": false,
2144
- "normalized": false,
2145
- "rstrip": false,
2146
- "single_word": false,
2147
- "special": true
2148
- },
2149
- "268": {
2150
- "content": "[extra_id_106]",
2151
- "lstrip": false,
2152
- "normalized": false,
2153
- "rstrip": false,
2154
- "single_word": false,
2155
- "special": true
2156
- },
2157
- "269": {
2158
- "content": "[extra_id_107]",
2159
- "lstrip": false,
2160
- "normalized": false,
2161
- "rstrip": false,
2162
- "single_word": false,
2163
- "special": true
2164
- },
2165
- "270": {
2166
- "content": "[extra_id_108]",
2167
- "lstrip": false,
2168
- "normalized": false,
2169
- "rstrip": false,
2170
- "single_word": false,
2171
- "special": true
2172
- },
2173
- "271": {
2174
- "content": "[extra_id_109]",
2175
- "lstrip": false,
2176
- "normalized": false,
2177
- "rstrip": false,
2178
- "single_word": false,
2179
- "special": true
2180
- },
2181
- "272": {
2182
- "content": "[extra_id_110]",
2183
- "lstrip": false,
2184
- "normalized": false,
2185
- "rstrip": false,
2186
- "single_word": false,
2187
- "special": true
2188
- },
2189
- "273": {
2190
- "content": "[extra_id_111]",
2191
- "lstrip": false,
2192
- "normalized": false,
2193
- "rstrip": false,
2194
- "single_word": false,
2195
- "special": true
2196
- },
2197
- "274": {
2198
- "content": "[extra_id_112]",
2199
- "lstrip": false,
2200
- "normalized": false,
2201
- "rstrip": false,
2202
- "single_word": false,
2203
- "special": true
2204
- },
2205
- "275": {
2206
- "content": "[extra_id_113]",
2207
- "lstrip": false,
2208
- "normalized": false,
2209
- "rstrip": false,
2210
- "single_word": false,
2211
- "special": true
2212
- },
2213
- "276": {
2214
- "content": "[extra_id_114]",
2215
- "lstrip": false,
2216
- "normalized": false,
2217
- "rstrip": false,
2218
- "single_word": false,
2219
- "special": true
2220
- },
2221
- "277": {
2222
- "content": "[extra_id_115]",
2223
- "lstrip": false,
2224
- "normalized": false,
2225
- "rstrip": false,
2226
- "single_word": false,
2227
- "special": true
2228
- },
2229
- "278": {
2230
- "content": "[extra_id_116]",
2231
- "lstrip": false,
2232
- "normalized": false,
2233
- "rstrip": false,
2234
- "single_word": false,
2235
- "special": true
2236
- },
2237
- "279": {
2238
- "content": "[extra_id_117]",
2239
- "lstrip": false,
2240
- "normalized": false,
2241
- "rstrip": false,
2242
- "single_word": false,
2243
- "special": true
2244
- },
2245
- "280": {
2246
- "content": "[extra_id_118]",
2247
- "lstrip": false,
2248
- "normalized": false,
2249
- "rstrip": false,
2250
- "single_word": false,
2251
- "special": true
2252
- },
2253
- "281": {
2254
- "content": "[extra_id_119]",
2255
- "lstrip": false,
2256
- "normalized": false,
2257
- "rstrip": false,
2258
- "single_word": false,
2259
- "special": true
2260
- },
2261
- "282": {
2262
- "content": "[extra_id_120]",
2263
- "lstrip": false,
2264
- "normalized": false,
2265
- "rstrip": false,
2266
- "single_word": false,
2267
- "special": true
2268
- },
2269
- "283": {
2270
- "content": "[extra_id_121]",
2271
- "lstrip": false,
2272
- "normalized": false,
2273
- "rstrip": false,
2274
- "single_word": false,
2275
- "special": true
2276
- },
2277
- "284": {
2278
- "content": "[extra_id_122]",
2279
- "lstrip": false,
2280
- "normalized": false,
2281
- "rstrip": false,
2282
- "single_word": false,
2283
- "special": true
2284
- },
2285
- "285": {
2286
- "content": "[extra_id_123]",
2287
- "lstrip": false,
2288
- "normalized": false,
2289
- "rstrip": false,
2290
- "single_word": false,
2291
- "special": true
2292
- },
2293
- "286": {
2294
- "content": "[extra_id_124]",
2295
- "lstrip": false,
2296
- "normalized": false,
2297
- "rstrip": false,
2298
- "single_word": false,
2299
- "special": true
2300
- },
2301
- "287": {
2302
- "content": "[extra_id_125]",
2303
- "lstrip": false,
2304
- "normalized": false,
2305
- "rstrip": false,
2306
- "single_word": false,
2307
- "special": true
2308
- },
2309
- "288": {
2310
- "content": "[extra_id_126]",
2311
- "lstrip": false,
2312
- "normalized": false,
2313
- "rstrip": false,
2314
- "single_word": false,
2315
- "special": true
2316
- },
2317
- "289": {
2318
- "content": "[extra_id_127]",
2319
- "lstrip": false,
2320
- "normalized": false,
2321
- "rstrip": false,
2322
- "single_word": false,
2323
- "special": true
2324
- },
2325
- "290": {
2326
- "content": "[extra_id_128]",
2327
- "lstrip": false,
2328
- "normalized": false,
2329
- "rstrip": false,
2330
- "single_word": false,
2331
- "special": true
2332
- },
2333
- "291": {
2334
- "content": "[extra_id_129]",
2335
- "lstrip": false,
2336
- "normalized": false,
2337
- "rstrip": false,
2338
- "single_word": false,
2339
- "special": true
2340
- },
2341
- "292": {
2342
- "content": "[extra_id_130]",
2343
- "lstrip": false,
2344
- "normalized": false,
2345
- "rstrip": false,
2346
- "single_word": false,
2347
- "special": true
2348
- },
2349
- "293": {
2350
- "content": "[extra_id_131]",
2351
- "lstrip": false,
2352
- "normalized": false,
2353
- "rstrip": false,
2354
- "single_word": false,
2355
- "special": true
2356
- },
2357
- "294": {
2358
- "content": "[extra_id_132]",
2359
- "lstrip": false,
2360
- "normalized": false,
2361
- "rstrip": false,
2362
- "single_word": false,
2363
- "special": true
2364
- },
2365
- "295": {
2366
- "content": "[extra_id_133]",
2367
- "lstrip": false,
2368
- "normalized": false,
2369
- "rstrip": false,
2370
- "single_word": false,
2371
- "special": true
2372
- },
2373
- "296": {
2374
- "content": "[extra_id_134]",
2375
- "lstrip": false,
2376
- "normalized": false,
2377
- "rstrip": false,
2378
- "single_word": false,
2379
- "special": true
2380
- },
2381
- "297": {
2382
- "content": "[extra_id_135]",
2383
- "lstrip": false,
2384
- "normalized": false,
2385
- "rstrip": false,
2386
- "single_word": false,
2387
- "special": true
2388
- },
2389
- "298": {
2390
- "content": "[extra_id_136]",
2391
- "lstrip": false,
2392
- "normalized": false,
2393
- "rstrip": false,
2394
- "single_word": false,
2395
- "special": true
2396
- },
2397
- "299": {
2398
- "content": "[extra_id_137]",
2399
- "lstrip": false,
2400
- "normalized": false,
2401
- "rstrip": false,
2402
- "single_word": false,
2403
- "special": true
2404
- },
2405
- "300": {
2406
- "content": "[extra_id_138]",
2407
- "lstrip": false,
2408
- "normalized": false,
2409
- "rstrip": false,
2410
- "single_word": false,
2411
- "special": true
2412
- },
2413
- "301": {
2414
- "content": "[extra_id_139]",
2415
- "lstrip": false,
2416
- "normalized": false,
2417
- "rstrip": false,
2418
- "single_word": false,
2419
- "special": true
2420
- },
2421
- "302": {
2422
- "content": "[extra_id_140]",
2423
- "lstrip": false,
2424
- "normalized": false,
2425
- "rstrip": false,
2426
- "single_word": false,
2427
- "special": true
2428
- },
2429
- "303": {
2430
- "content": "[extra_id_141]",
2431
- "lstrip": false,
2432
- "normalized": false,
2433
- "rstrip": false,
2434
- "single_word": false,
2435
- "special": true
2436
- },
2437
- "304": {
2438
- "content": "[extra_id_142]",
2439
- "lstrip": false,
2440
- "normalized": false,
2441
- "rstrip": false,
2442
- "single_word": false,
2443
- "special": true
2444
- },
2445
- "305": {
2446
- "content": "[extra_id_143]",
2447
- "lstrip": false,
2448
- "normalized": false,
2449
- "rstrip": false,
2450
- "single_word": false,
2451
- "special": true
2452
- },
2453
- "306": {
2454
- "content": "[extra_id_144]",
2455
- "lstrip": false,
2456
- "normalized": false,
2457
- "rstrip": false,
2458
- "single_word": false,
2459
- "special": true
2460
- },
2461
- "307": {
2462
- "content": "[extra_id_145]",
2463
- "lstrip": false,
2464
- "normalized": false,
2465
- "rstrip": false,
2466
- "single_word": false,
2467
- "special": true
2468
- },
2469
- "308": {
2470
- "content": "[extra_id_146]",
2471
- "lstrip": false,
2472
- "normalized": false,
2473
- "rstrip": false,
2474
- "single_word": false,
2475
- "special": true
2476
- },
2477
- "309": {
2478
- "content": "[extra_id_147]",
2479
- "lstrip": false,
2480
- "normalized": false,
2481
- "rstrip": false,
2482
- "single_word": false,
2483
- "special": true
2484
- },
2485
- "310": {
2486
- "content": "[extra_id_148]",
2487
- "lstrip": false,
2488
- "normalized": false,
2489
- "rstrip": false,
2490
- "single_word": false,
2491
- "special": true
2492
- },
2493
- "311": {
2494
- "content": "[extra_id_149]",
2495
- "lstrip": false,
2496
- "normalized": false,
2497
- "rstrip": false,
2498
- "single_word": false,
2499
- "special": true
2500
- },
2501
- "312": {
2502
- "content": "[extra_id_150]",
2503
- "lstrip": false,
2504
- "normalized": false,
2505
- "rstrip": false,
2506
- "single_word": false,
2507
- "special": true
2508
- },
2509
- "313": {
2510
- "content": "[extra_id_151]",
2511
- "lstrip": false,
2512
- "normalized": false,
2513
- "rstrip": false,
2514
- "single_word": false,
2515
- "special": true
2516
- },
2517
- "314": {
2518
- "content": "[extra_id_152]",
2519
- "lstrip": false,
2520
- "normalized": false,
2521
- "rstrip": false,
2522
- "single_word": false,
2523
- "special": true
2524
- },
2525
- "315": {
2526
- "content": "[extra_id_153]",
2527
- "lstrip": false,
2528
- "normalized": false,
2529
- "rstrip": false,
2530
- "single_word": false,
2531
- "special": true
2532
- },
2533
- "316": {
2534
- "content": "[extra_id_154]",
2535
- "lstrip": false,
2536
- "normalized": false,
2537
- "rstrip": false,
2538
- "single_word": false,
2539
- "special": true
2540
- },
2541
- "317": {
2542
- "content": "[extra_id_155]",
2543
- "lstrip": false,
2544
- "normalized": false,
2545
- "rstrip": false,
2546
- "single_word": false,
2547
- "special": true
2548
- },
2549
- "318": {
2550
- "content": "[extra_id_156]",
2551
- "lstrip": false,
2552
- "normalized": false,
2553
- "rstrip": false,
2554
- "single_word": false,
2555
- "special": true
2556
- },
2557
- "319": {
2558
- "content": "[extra_id_157]",
2559
- "lstrip": false,
2560
- "normalized": false,
2561
- "rstrip": false,
2562
- "single_word": false,
2563
- "special": true
2564
- },
2565
- "320": {
2566
- "content": "[extra_id_158]",
2567
- "lstrip": false,
2568
- "normalized": false,
2569
- "rstrip": false,
2570
- "single_word": false,
2571
- "special": true
2572
- },
2573
- "321": {
2574
- "content": "[extra_id_159]",
2575
- "lstrip": false,
2576
- "normalized": false,
2577
- "rstrip": false,
2578
- "single_word": false,
2579
- "special": true
2580
- },
2581
- "322": {
2582
- "content": "[extra_id_160]",
2583
- "lstrip": false,
2584
- "normalized": false,
2585
- "rstrip": false,
2586
- "single_word": false,
2587
- "special": true
2588
- },
2589
- "323": {
2590
- "content": "[extra_id_161]",
2591
- "lstrip": false,
2592
- "normalized": false,
2593
- "rstrip": false,
2594
- "single_word": false,
2595
- "special": true
2596
- },
2597
- "324": {
2598
- "content": "[extra_id_162]",
2599
- "lstrip": false,
2600
- "normalized": false,
2601
- "rstrip": false,
2602
- "single_word": false,
2603
- "special": true
2604
- },
2605
- "325": {
2606
- "content": "[extra_id_163]",
2607
- "lstrip": false,
2608
- "normalized": false,
2609
- "rstrip": false,
2610
- "single_word": false,
2611
- "special": true
2612
- },
2613
- "326": {
2614
- "content": "[extra_id_164]",
2615
- "lstrip": false,
2616
- "normalized": false,
2617
- "rstrip": false,
2618
- "single_word": false,
2619
- "special": true
2620
- },
2621
- "327": {
2622
- "content": "[extra_id_165]",
2623
- "lstrip": false,
2624
- "normalized": false,
2625
- "rstrip": false,
2626
- "single_word": false,
2627
- "special": true
2628
- },
2629
- "328": {
2630
- "content": "[extra_id_166]",
2631
- "lstrip": false,
2632
- "normalized": false,
2633
- "rstrip": false,
2634
- "single_word": false,
2635
- "special": true
2636
- },
2637
- "329": {
2638
- "content": "[extra_id_167]",
2639
- "lstrip": false,
2640
- "normalized": false,
2641
- "rstrip": false,
2642
- "single_word": false,
2643
- "special": true
2644
- },
2645
- "330": {
2646
- "content": "[extra_id_168]",
2647
- "lstrip": false,
2648
- "normalized": false,
2649
- "rstrip": false,
2650
- "single_word": false,
2651
- "special": true
2652
- },
2653
- "331": {
2654
- "content": "[extra_id_169]",
2655
- "lstrip": false,
2656
- "normalized": false,
2657
- "rstrip": false,
2658
- "single_word": false,
2659
- "special": true
2660
- },
2661
- "332": {
2662
- "content": "[extra_id_170]",
2663
- "lstrip": false,
2664
- "normalized": false,
2665
- "rstrip": false,
2666
- "single_word": false,
2667
- "special": true
2668
- },
2669
- "333": {
2670
- "content": "[extra_id_171]",
2671
- "lstrip": false,
2672
- "normalized": false,
2673
- "rstrip": false,
2674
- "single_word": false,
2675
- "special": true
2676
- },
2677
- "334": {
2678
- "content": "[extra_id_172]",
2679
- "lstrip": false,
2680
- "normalized": false,
2681
- "rstrip": false,
2682
- "single_word": false,
2683
- "special": true
2684
- },
2685
- "335": {
2686
- "content": "[extra_id_173]",
2687
- "lstrip": false,
2688
- "normalized": false,
2689
- "rstrip": false,
2690
- "single_word": false,
2691
- "special": true
2692
- },
2693
- "336": {
2694
- "content": "[extra_id_174]",
2695
- "lstrip": false,
2696
- "normalized": false,
2697
- "rstrip": false,
2698
- "single_word": false,
2699
- "special": true
2700
- },
2701
- "337": {
2702
- "content": "[extra_id_175]",
2703
- "lstrip": false,
2704
- "normalized": false,
2705
- "rstrip": false,
2706
- "single_word": false,
2707
- "special": true
2708
- },
2709
- "338": {
2710
- "content": "[extra_id_176]",
2711
- "lstrip": false,
2712
- "normalized": false,
2713
- "rstrip": false,
2714
- "single_word": false,
2715
- "special": true
2716
- },
2717
- "339": {
2718
- "content": "[extra_id_177]",
2719
- "lstrip": false,
2720
- "normalized": false,
2721
- "rstrip": false,
2722
- "single_word": false,
2723
- "special": true
2724
- },
2725
- "340": {
2726
- "content": "[extra_id_178]",
2727
- "lstrip": false,
2728
- "normalized": false,
2729
- "rstrip": false,
2730
- "single_word": false,
2731
- "special": true
2732
- },
2733
- "341": {
2734
- "content": "[extra_id_179]",
2735
- "lstrip": false,
2736
- "normalized": false,
2737
- "rstrip": false,
2738
- "single_word": false,
2739
- "special": true
2740
- },
2741
- "342": {
2742
- "content": "[extra_id_180]",
2743
- "lstrip": false,
2744
- "normalized": false,
2745
- "rstrip": false,
2746
- "single_word": false,
2747
- "special": true
2748
- },
2749
- "343": {
2750
- "content": "[extra_id_181]",
2751
- "lstrip": false,
2752
- "normalized": false,
2753
- "rstrip": false,
2754
- "single_word": false,
2755
- "special": true
2756
- },
2757
- "344": {
2758
- "content": "[extra_id_182]",
2759
- "lstrip": false,
2760
- "normalized": false,
2761
- "rstrip": false,
2762
- "single_word": false,
2763
- "special": true
2764
- },
2765
- "345": {
2766
- "content": "[extra_id_183]",
2767
- "lstrip": false,
2768
- "normalized": false,
2769
- "rstrip": false,
2770
- "single_word": false,
2771
- "special": true
2772
- },
2773
- "346": {
2774
- "content": "[extra_id_184]",
2775
- "lstrip": false,
2776
- "normalized": false,
2777
- "rstrip": false,
2778
- "single_word": false,
2779
- "special": true
2780
- },
2781
- "347": {
2782
- "content": "[extra_id_185]",
2783
- "lstrip": false,
2784
- "normalized": false,
2785
- "rstrip": false,
2786
- "single_word": false,
2787
- "special": true
2788
- },
2789
- "348": {
2790
- "content": "[extra_id_186]",
2791
- "lstrip": false,
2792
- "normalized": false,
2793
- "rstrip": false,
2794
- "single_word": false,
2795
- "special": true
2796
- },
2797
- "349": {
2798
- "content": "[extra_id_187]",
2799
- "lstrip": false,
2800
- "normalized": false,
2801
- "rstrip": false,
2802
- "single_word": false,
2803
- "special": true
2804
- },
2805
- "350": {
2806
- "content": "[extra_id_188]",
2807
- "lstrip": false,
2808
- "normalized": false,
2809
- "rstrip": false,
2810
- "single_word": false,
2811
- "special": true
2812
- },
2813
- "351": {
2814
- "content": "</tool_call>",
2815
- "lstrip": false,
2816
- "normalized": false,
2817
- "rstrip": false,
2818
- "single_word": false,
2819
- "special": false
2820
- },
2821
- "352": {
2822
- "content": "<tool_call>",
2823
- "lstrip": false,
2824
- "normalized": false,
2825
- "rstrip": false,
2826
- "single_word": false,
2827
- "special": false
2828
- },
2829
- "353": {
2830
- "content": "</tool_result>",
2831
- "lstrip": false,
2832
- "normalized": false,
2833
- "rstrip": false,
2834
- "single_word": false,
2835
- "special": false
2836
- },
2837
- "354": {
2838
- "content": "<tool_result>",
2839
- "lstrip": false,
2840
- "normalized": false,
2841
- "rstrip": false,
2842
- "single_word": false,
2843
- "special": false
2844
- },
2845
- "355": {
2846
- "content": "</think>",
2847
- "lstrip": false,
2848
- "normalized": false,
2849
- "rstrip": false,
2850
- "single_word": false,
2851
- "special": false
2852
- },
2853
- "356": {
2854
- "content": "<think>",
2855
- "lstrip": false,
2856
- "normalized": false,
2857
- "rstrip": false,
2858
- "single_word": false,
2859
- "special": false
2860
- },
2861
- "357": {
2862
- "content": "[|system|]",
2863
- "lstrip": false,
2864
- "normalized": false,
2865
- "rstrip": false,
2866
- "single_word": false,
2867
- "special": true
2868
- },
2869
- "358": {
2870
- "content": "[|tool|]",
2871
- "lstrip": false,
2872
- "normalized": false,
2873
- "rstrip": false,
2874
- "single_word": false,
2875
- "special": true
2876
- },
2877
- "359": {
2878
- "content": "[|assistant|]",
2879
- "lstrip": false,
2880
- "normalized": false,
2881
- "rstrip": false,
2882
- "single_word": false,
2883
- "special": true
2884
- },
2885
- "360": {
2886
- "content": "[|user|]",
2887
- "lstrip": false,
2888
- "normalized": false,
2889
- "rstrip": false,
2890
- "single_word": false,
2891
- "special": true
2892
- },
2893
- "361": {
2894
- "content": "[|endofturn|]",
2895
- "lstrip": false,
2896
- "normalized": false,
2897
- "rstrip": false,
2898
- "single_word": false,
2899
- "special": true
2900
- }
2901
- },
2902
- "additional_special_tokens": [
2903
- "[unused0]",
2904
- "[unused1]",
2905
- "[unused2]",
2906
- "[unused3]",
2907
- "[unused4]",
2908
- "[unused5]",
2909
- "[unused6]",
2910
- "[unused7]",
2911
- "[unused8]",
2912
- "[unused9]",
2913
- "[unused10]",
2914
- "[unused11]",
2915
- "[unused12]",
2916
- "[unused13]",
2917
- "[unused14]",
2918
- "[unused15]",
2919
- "[unused16]",
2920
- "[unused17]",
2921
- "[unused18]",
2922
- "[unused19]",
2923
- "[unused20]",
2924
- "[unused21]",
2925
- "[unused22]",
2926
- "[unused23]",
2927
- "[unused24]",
2928
- "[unused25]",
2929
- "[unused26]",
2930
- "[unused27]",
2931
- "[unused28]",
2932
- "[unused29]",
2933
- "[unused30]",
2934
- "[unused31]",
2935
- "[unused32]",
2936
- "[unused33]",
2937
- "[unused34]",
2938
- "[unused35]",
2939
- "[unused36]",
2940
- "[unused37]",
2941
- "[unused38]",
2942
- "[unused39]",
2943
- "[unused40]",
2944
- "[unused41]",
2945
- "[unused42]",
2946
- "[unused43]",
2947
- "[unused44]",
2948
- "[unused45]",
2949
- "[unused46]",
2950
- "[unused47]",
2951
- "[unused48]",
2952
- "[unused49]",
2953
- "[unused50]",
2954
- "[unused51]",
2955
- "[unused52]",
2956
- "[unused53]",
2957
- "[unused54]",
2958
- "[unused55]",
2959
- "[unused56]",
2960
- "[unused57]",
2961
- "[unused58]",
2962
- "[unused59]",
2963
- "[unused60]",
2964
- "[unused61]",
2965
- "[unused62]",
2966
- "[unused63]",
2967
- "[unused64]",
2968
- "[unused65]",
2969
- "[unused66]",
2970
- "[unused67]",
2971
- "[unused68]",
2972
- "[unused69]",
2973
- "[unused70]",
2974
- "[unused71]",
2975
- "[unused72]",
2976
- "[unused73]",
2977
- "[unused74]",
2978
- "[unused75]",
2979
- "[unused76]",
2980
- "[unused77]",
2981
- "[unused78]",
2982
- "[unused79]",
2983
- "[unused80]",
2984
- "[unused81]",
2985
- "[unused82]",
2986
- "[unused83]",
2987
- "[unused84]",
2988
- "[unused85]",
2989
- "[unused86]",
2990
- "[unused87]",
2991
- "[unused88]",
2992
- "[unused89]",
2993
- "[unused90]",
2994
- "[unused91]",
2995
- "[unused92]",
2996
- "[unused93]",
2997
- "[unused94]",
2998
- "[unused95]",
2999
- "[unused96]",
3000
- "[unused97]",
3001
- "[unused98]",
3002
- "[unused99]",
3003
- "[extra_id_0]",
3004
- "[extra_id_1]",
3005
- "[extra_id_2]",
3006
- "[extra_id_3]",
3007
- "[extra_id_4]",
3008
- "[extra_id_5]",
3009
- "[extra_id_6]",
3010
- "[extra_id_7]",
3011
- "[extra_id_8]",
3012
- "[extra_id_9]",
3013
- "[extra_id_10]",
3014
- "[extra_id_11]",
3015
- "[extra_id_12]",
3016
- "[extra_id_13]",
3017
- "[extra_id_14]",
3018
- "[extra_id_15]",
3019
- "[extra_id_16]",
3020
- "[extra_id_17]",
3021
- "[extra_id_18]",
3022
- "[extra_id_19]",
3023
- "[extra_id_20]",
3024
- "[extra_id_21]",
3025
- "[extra_id_22]",
3026
- "[extra_id_23]",
3027
- "[extra_id_24]",
3028
- "[extra_id_25]",
3029
- "[extra_id_26]",
3030
- "[extra_id_27]",
3031
- "[extra_id_28]",
3032
- "[extra_id_29]",
3033
- "[extra_id_30]",
3034
- "[extra_id_31]",
3035
- "[extra_id_32]",
3036
- "[extra_id_33]",
3037
- "[extra_id_34]",
3038
- "[extra_id_35]",
3039
- "[extra_id_36]",
3040
- "[extra_id_37]",
3041
- "[extra_id_38]",
3042
- "[extra_id_39]",
3043
- "[extra_id_40]",
3044
- "[extra_id_41]",
3045
- "[extra_id_42]",
3046
- "[extra_id_43]",
3047
- "[extra_id_44]",
3048
- "[extra_id_45]",
3049
- "[extra_id_46]",
3050
- "[extra_id_47]",
3051
- "[extra_id_48]",
3052
- "[extra_id_49]",
3053
- "[extra_id_50]",
3054
- "[extra_id_51]",
3055
- "[extra_id_52]",
3056
- "[extra_id_53]",
3057
- "[extra_id_54]",
3058
- "[extra_id_55]",
3059
- "[extra_id_56]",
3060
- "[extra_id_57]",
3061
- "[extra_id_58]",
3062
- "[extra_id_59]",
3063
- "[extra_id_60]",
3064
- "[extra_id_61]",
3065
- "[extra_id_62]",
3066
- "[extra_id_63]",
3067
- "[extra_id_64]",
3068
- "[extra_id_65]",
3069
- "[extra_id_66]",
3070
- "[extra_id_67]",
3071
- "[extra_id_68]",
3072
- "[extra_id_69]",
3073
- "[extra_id_70]",
3074
- "[extra_id_71]",
3075
- "[extra_id_72]",
3076
- "[extra_id_73]",
3077
- "[extra_id_74]",
3078
- "[extra_id_75]",
3079
- "[extra_id_76]",
3080
- "[extra_id_77]",
3081
- "[extra_id_78]",
3082
- "[extra_id_79]",
3083
- "[extra_id_80]",
3084
- "[extra_id_81]",
3085
- "[extra_id_82]",
3086
- "[extra_id_83]",
3087
- "[extra_id_84]",
3088
- "[extra_id_85]",
3089
- "[extra_id_86]",
3090
- "[extra_id_87]",
3091
- "[extra_id_88]",
3092
- "[extra_id_89]",
3093
- "[extra_id_90]",
3094
- "[extra_id_91]",
3095
- "[extra_id_92]",
3096
- "[extra_id_93]",
3097
- "[extra_id_94]",
3098
- "[extra_id_95]",
3099
- "[extra_id_96]",
3100
- "[extra_id_97]",
3101
- "[extra_id_98]",
3102
- "[extra_id_99]",
3103
- "[extra_id_100]",
3104
- "[extra_id_101]",
3105
- "[extra_id_102]",
3106
- "[extra_id_103]",
3107
- "[extra_id_104]",
3108
- "[extra_id_105]",
3109
- "[extra_id_106]",
3110
- "[extra_id_107]",
3111
- "[extra_id_108]",
3112
- "[extra_id_109]",
3113
- "[extra_id_110]",
3114
- "[extra_id_111]",
3115
- "[extra_id_112]",
3116
- "[extra_id_113]",
3117
- "[extra_id_114]",
3118
- "[extra_id_115]",
3119
- "[extra_id_116]",
3120
- "[extra_id_117]",
3121
- "[extra_id_118]",
3122
- "[extra_id_119]",
3123
- "[extra_id_120]",
3124
- "[extra_id_121]",
3125
- "[extra_id_122]",
3126
- "[extra_id_123]",
3127
- "[extra_id_124]",
3128
- "[extra_id_125]",
3129
- "[extra_id_126]",
3130
- "[extra_id_127]",
3131
- "[extra_id_128]",
3132
- "[extra_id_129]",
3133
- "[extra_id_130]",
3134
- "[extra_id_131]",
3135
- "[extra_id_132]",
3136
- "[extra_id_133]",
3137
- "[extra_id_134]",
3138
- "[extra_id_135]",
3139
- "[extra_id_136]",
3140
- "[extra_id_137]",
3141
- "[extra_id_138]",
3142
- "[extra_id_139]",
3143
- "[extra_id_140]",
3144
- "[extra_id_141]",
3145
- "[extra_id_142]",
3146
- "[extra_id_143]",
3147
- "[extra_id_144]",
3148
- "[extra_id_145]",
3149
- "[extra_id_146]",
3150
- "[extra_id_147]",
3151
- "[extra_id_148]",
3152
- "[extra_id_149]",
3153
- "[extra_id_150]",
3154
- "[extra_id_151]",
3155
- "[extra_id_152]",
3156
- "[extra_id_153]",
3157
- "[extra_id_154]",
3158
- "[extra_id_155]",
3159
- "[extra_id_156]",
3160
- "[extra_id_157]",
3161
- "[extra_id_158]",
3162
- "[extra_id_159]",
3163
- "[extra_id_160]",
3164
- "[extra_id_161]",
3165
- "[extra_id_162]",
3166
- "[extra_id_163]",
3167
- "[extra_id_164]",
3168
- "[extra_id_165]",
3169
- "[extra_id_166]",
3170
- "[extra_id_167]",
3171
- "[extra_id_168]",
3172
- "[extra_id_169]",
3173
- "[extra_id_170]",
3174
- "[extra_id_171]",
3175
- "[extra_id_172]",
3176
- "[extra_id_173]",
3177
- "[extra_id_174]",
3178
- "[extra_id_175]",
3179
- "[extra_id_176]",
3180
- "[extra_id_177]",
3181
- "[extra_id_178]",
3182
- "[extra_id_179]",
3183
- "[extra_id_180]",
3184
- "[extra_id_181]",
3185
- "[extra_id_182]",
3186
- "[extra_id_183]",
3187
- "[extra_id_184]",
3188
- "[extra_id_185]",
3189
- "[extra_id_186]",
3190
- "[extra_id_187]",
3191
- "[extra_id_188]",
3192
- "[|system|]",
3193
- "[|tool|]",
3194
- "[|assistant|]",
3195
- "[|user|]",
3196
- "[|endofturn|]",
3197
- "PI:URL",
3198
- "PI:EMAIL",
3199
- "PI:ACCOUNT_NUM",
3200
- "PI:PHONE_NUM",
3201
- "PI:BUSINESS_NUM",
3202
- "PI:ANNON",
3203
- "PI:KEY",
3204
- "PI:ID",
3205
- "PI:IP_ADDRESS",
3206
- "PI:USER"
3207
- ],
3208
- "bos_token": "[BOS]",
3209
- "clean_up_tokenization_spaces": false,
3210
- "eos_token": "[|endofturn|]",
3211
- "errors": "replace",
3212
- "extra_special_tokens": {},
3213
- "model_max_length": 2048,
3214
- "pad_token": "[PAD]",
3215
- "padding_side": "right",
3216
- "split_special_tokens": false,
3217
- "tokenizer_class": "GPT2Tokenizer",
3218
- "unk_token": "[UNK]"
3219
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
checkpoint-6981/trainer_state.json DELETED
The diff for this file is too large to render. See raw diff
 
checkpoint-6981/training_args.bin DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:7bdf0b610d3ed816fddd1b8b7973a6a2e0f5ff23810759e30d6b2b2edaef0712
3
- size 7697
 
 
 
 
checkpoint-6981/vocab.json DELETED
The diff for this file is too large to render. See raw diff
 
checkpoint-6981/zero_to_fp32.py DELETED
@@ -1,604 +0,0 @@
1
- #!/usr/bin/env python
2
-
3
- # Copyright (c) Microsoft Corporation.
4
- # SPDX-License-Identifier: Apache-2.0
5
-
6
- # DeepSpeed Team
7
-
8
- # This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
9
- # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
- # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
- # application.
12
- #
13
- # example: python zero_to_fp32.py . pytorch_model.bin
14
-
15
- import argparse
16
- import torch
17
- import glob
18
- import math
19
- import os
20
- import re
21
- from collections import OrderedDict
22
- from dataclasses import dataclass
23
-
24
- # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
25
- # DeepSpeed data structures it has to be available in the current python environment.
26
- from deepspeed.utils import logger
27
- from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
28
- FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
29
- FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
30
-
31
-
32
- @dataclass
33
- class zero_model_state:
34
- buffers: dict()
35
- param_shapes: dict()
36
- shared_params: list
37
- ds_version: int
38
- frozen_param_shapes: dict()
39
- frozen_param_fragments: dict()
40
-
41
-
42
- debug = 0
43
-
44
- # load to cpu
45
- device = torch.device('cpu')
46
-
47
-
48
- def atoi(text):
49
- return int(text) if text.isdigit() else text
50
-
51
-
52
- def natural_keys(text):
53
- '''
54
- alist.sort(key=natural_keys) sorts in human order
55
- http://nedbatchelder.com/blog/200712/human_sorting.html
56
- (See Toothy's implementation in the comments)
57
- '''
58
- return [atoi(c) for c in re.split(r'(\d+)', text)]
59
-
60
-
61
- def get_model_state_file(checkpoint_dir, zero_stage):
62
- if not os.path.isdir(checkpoint_dir):
63
- raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
64
-
65
- # there should be only one file
66
- if zero_stage <= 2:
67
- file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
68
- elif zero_stage == 3:
69
- file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
70
-
71
- if not os.path.exists(file):
72
- raise FileNotFoundError(f"can't find model states file at '{file}'")
73
-
74
- return file
75
-
76
-
77
- def get_checkpoint_files(checkpoint_dir, glob_pattern):
78
- # XXX: need to test that this simple glob rule works for multi-node setup too
79
- ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
80
-
81
- if len(ckpt_files) == 0:
82
- raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
83
-
84
- return ckpt_files
85
-
86
-
87
- def get_optim_files(checkpoint_dir):
88
- return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
89
-
90
-
91
- def get_model_state_files(checkpoint_dir):
92
- return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
93
-
94
-
95
- def parse_model_states(files):
96
- zero_model_states = []
97
- for file in files:
98
- state_dict = torch.load(file, map_location=device)
99
-
100
- if BUFFER_NAMES not in state_dict:
101
- raise ValueError(f"{file} is not a model state checkpoint")
102
- buffer_names = state_dict[BUFFER_NAMES]
103
- if debug:
104
- print("Found buffers:", buffer_names)
105
-
106
- # recover just the buffers while restoring them to fp32 if they were saved in fp16
107
- buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
108
- param_shapes = state_dict[PARAM_SHAPES]
109
-
110
- # collect parameters that are included in param_shapes
111
- param_names = []
112
- for s in param_shapes:
113
- for name in s.keys():
114
- param_names.append(name)
115
-
116
- # update with frozen parameters
117
- frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
118
- if frozen_param_shapes is not None:
119
- if debug:
120
- print(f"Found frozen_param_shapes: {frozen_param_shapes}")
121
- param_names += list(frozen_param_shapes.keys())
122
-
123
- # handle shared params
124
- shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
125
-
126
- ds_version = state_dict.get(DS_VERSION, None)
127
-
128
- frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
129
-
130
- z_model_state = zero_model_state(buffers=buffers,
131
- param_shapes=param_shapes,
132
- shared_params=shared_params,
133
- ds_version=ds_version,
134
- frozen_param_shapes=frozen_param_shapes,
135
- frozen_param_fragments=frozen_param_fragments)
136
- zero_model_states.append(z_model_state)
137
-
138
- return zero_model_states
139
-
140
-
141
- def parse_optim_states(files, ds_checkpoint_dir):
142
-
143
- total_files = len(files)
144
- state_dicts = []
145
- for f in files:
146
- state_dict = torch.load(f, map_location=device)
147
- # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
148
- # and also handle the case where it was already removed by another helper script
149
- state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
150
- state_dicts.append(state_dict)
151
-
152
- if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
153
- raise ValueError(f"{files[0]} is not a zero checkpoint")
154
- zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
155
- world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
156
-
157
- # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
158
- # parameters can be different from data parallelism for non-expert parameters. So we can just
159
- # use the max of the partition_count to get the dp world_size.
160
-
161
- if type(world_size) is list:
162
- world_size = max(world_size)
163
-
164
- if world_size != total_files:
165
- raise ValueError(
166
- f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
167
- "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
168
- )
169
-
170
- # the groups are named differently in each stage
171
- if zero_stage <= 2:
172
- fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
173
- elif zero_stage == 3:
174
- fp32_groups_key = FP32_FLAT_GROUPS
175
- else:
176
- raise ValueError(f"unknown zero stage {zero_stage}")
177
-
178
- if zero_stage <= 2:
179
- fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
180
- elif zero_stage == 3:
181
- # if there is more than one param group, there will be multiple flattened tensors - one
182
- # flattened tensor per group - for simplicity merge them into a single tensor
183
- #
184
- # XXX: could make the script more memory efficient for when there are multiple groups - it
185
- # will require matching the sub-lists of param_shapes for each param group flattened tensor
186
-
187
- fp32_flat_groups = [
188
- torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
189
- ]
190
-
191
- return zero_stage, world_size, fp32_flat_groups
192
-
193
-
194
- def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
195
- """
196
- Returns fp32 state_dict reconstructed from ds checkpoint
197
-
198
- Args:
199
- - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
200
-
201
- """
202
- print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
203
-
204
- optim_files = get_optim_files(ds_checkpoint_dir)
205
- zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
206
- print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
207
-
208
- model_files = get_model_state_files(ds_checkpoint_dir)
209
-
210
- zero_model_states = parse_model_states(model_files)
211
- print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
212
-
213
- if zero_stage <= 2:
214
- return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
215
- exclude_frozen_parameters)
216
- elif zero_stage == 3:
217
- return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
218
- exclude_frozen_parameters)
219
-
220
-
221
- def _zero2_merge_frozen_params(state_dict, zero_model_states):
222
- if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
223
- return
224
-
225
- frozen_param_shapes = zero_model_states[0].frozen_param_shapes
226
- frozen_param_fragments = zero_model_states[0].frozen_param_fragments
227
-
228
- if debug:
229
- num_elem = sum(s.numel() for s in frozen_param_shapes.values())
230
- print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
231
-
232
- wanted_params = len(frozen_param_shapes)
233
- wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
234
- avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
235
- print(f'Frozen params: Have {avail_numel} numels to process.')
236
- print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
237
-
238
- total_params = 0
239
- total_numel = 0
240
- for name, shape in frozen_param_shapes.items():
241
- total_params += 1
242
- unpartitioned_numel = shape.numel()
243
- total_numel += unpartitioned_numel
244
-
245
- state_dict[name] = frozen_param_fragments[name]
246
-
247
- if debug:
248
- print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
249
-
250
- print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
251
-
252
-
253
- def _has_callable(obj, fn):
254
- attr = getattr(obj, fn, None)
255
- return callable(attr)
256
-
257
-
258
- def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
259
- param_shapes = zero_model_states[0].param_shapes
260
-
261
- # Reconstruction protocol:
262
- #
263
- # XXX: document this
264
-
265
- if debug:
266
- for i in range(world_size):
267
- for j in range(len(fp32_flat_groups[0])):
268
- print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
269
-
270
- # XXX: memory usage doubles here (zero2)
271
- num_param_groups = len(fp32_flat_groups[0])
272
- merged_single_partition_of_fp32_groups = []
273
- for i in range(num_param_groups):
274
- merged_partitions = [sd[i] for sd in fp32_flat_groups]
275
- full_single_fp32_vector = torch.cat(merged_partitions, 0)
276
- merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
277
- avail_numel = sum(
278
- [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
279
-
280
- if debug:
281
- wanted_params = sum([len(shapes) for shapes in param_shapes])
282
- wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
283
- # not asserting if there is a mismatch due to possible padding
284
- print(f"Have {avail_numel} numels to process.")
285
- print(f"Need {wanted_numel} numels in {wanted_params} params.")
286
-
287
- # params
288
- # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
289
- # out-of-core computing solution
290
- total_numel = 0
291
- total_params = 0
292
- for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
293
- offset = 0
294
- avail_numel = full_single_fp32_vector.numel()
295
- for name, shape in shapes.items():
296
-
297
- unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
298
- total_numel += unpartitioned_numel
299
- total_params += 1
300
-
301
- if debug:
302
- print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
303
- state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
304
- offset += unpartitioned_numel
305
-
306
- # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
307
- # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
308
- # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
309
- # live optimizer object, so we are checking that the numbers are within the right range
310
- align_to = 2 * world_size
311
-
312
- def zero2_align(x):
313
- return align_to * math.ceil(x / align_to)
314
-
315
- if debug:
316
- print(f"original offset={offset}, avail_numel={avail_numel}")
317
-
318
- offset = zero2_align(offset)
319
- avail_numel = zero2_align(avail_numel)
320
-
321
- if debug:
322
- print(f"aligned offset={offset}, avail_numel={avail_numel}")
323
-
324
- # Sanity check
325
- if offset != avail_numel:
326
- raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
327
-
328
- print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
329
-
330
-
331
- def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
332
- exclude_frozen_parameters):
333
- state_dict = OrderedDict()
334
-
335
- # buffers
336
- buffers = zero_model_states[0].buffers
337
- state_dict.update(buffers)
338
- if debug:
339
- print(f"added {len(buffers)} buffers")
340
-
341
- if not exclude_frozen_parameters:
342
- _zero2_merge_frozen_params(state_dict, zero_model_states)
343
-
344
- _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
345
-
346
- # recover shared parameters
347
- for pair in zero_model_states[0].shared_params:
348
- if pair[1] in state_dict:
349
- state_dict[pair[0]] = state_dict[pair[1]]
350
-
351
- return state_dict
352
-
353
-
354
- def zero3_partitioned_param_info(unpartitioned_numel, world_size):
355
- remainder = unpartitioned_numel % world_size
356
- padding_numel = (world_size - remainder) if remainder else 0
357
- partitioned_numel = math.ceil(unpartitioned_numel / world_size)
358
- return partitioned_numel, padding_numel
359
-
360
-
361
- def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
362
- if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
363
- return
364
-
365
- if debug:
366
- for i in range(world_size):
367
- num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
368
- print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
369
-
370
- frozen_param_shapes = zero_model_states[0].frozen_param_shapes
371
- wanted_params = len(frozen_param_shapes)
372
- wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
373
- avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
374
- print(f'Frozen params: Have {avail_numel} numels to process.')
375
- print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
376
-
377
- total_params = 0
378
- total_numel = 0
379
- for name, shape in zero_model_states[0].frozen_param_shapes.items():
380
- total_params += 1
381
- unpartitioned_numel = shape.numel()
382
- total_numel += unpartitioned_numel
383
-
384
- param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
385
- state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
386
-
387
- partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
388
-
389
- if debug:
390
- print(
391
- f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
392
- )
393
-
394
- print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
395
-
396
-
397
- def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
398
- param_shapes = zero_model_states[0].param_shapes
399
- avail_numel = fp32_flat_groups[0].numel() * world_size
400
- # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
401
- # param, re-consolidating each param, while dealing with padding if any
402
-
403
- # merge list of dicts, preserving order
404
- param_shapes = {k: v for d in param_shapes for k, v in d.items()}
405
-
406
- if debug:
407
- for i in range(world_size):
408
- print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
409
-
410
- wanted_params = len(param_shapes)
411
- wanted_numel = sum(shape.numel() for shape in param_shapes.values())
412
- # not asserting if there is a mismatch due to possible padding
413
- avail_numel = fp32_flat_groups[0].numel() * world_size
414
- print(f"Trainable params: Have {avail_numel} numels to process.")
415
- print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
416
-
417
- # params
418
- # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
419
- # out-of-core computing solution
420
- offset = 0
421
- total_numel = 0
422
- total_params = 0
423
- for name, shape in param_shapes.items():
424
-
425
- unpartitioned_numel = shape.numel()
426
- total_numel += unpartitioned_numel
427
- total_params += 1
428
-
429
- partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
430
-
431
- if debug:
432
- print(
433
- f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
434
- )
435
-
436
- # XXX: memory usage doubles here
437
- state_dict[name] = torch.cat(
438
- tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
439
- 0).narrow(0, 0, unpartitioned_numel).view(shape)
440
- offset += partitioned_numel
441
-
442
- offset *= world_size
443
-
444
- # Sanity check
445
- if offset != avail_numel:
446
- raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
447
-
448
- print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
449
-
450
-
451
- def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
452
- exclude_frozen_parameters):
453
- state_dict = OrderedDict()
454
-
455
- # buffers
456
- buffers = zero_model_states[0].buffers
457
- state_dict.update(buffers)
458
- if debug:
459
- print(f"added {len(buffers)} buffers")
460
-
461
- if not exclude_frozen_parameters:
462
- _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
463
-
464
- _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
465
-
466
- # recover shared parameters
467
- for pair in zero_model_states[0].shared_params:
468
- if pair[1] in state_dict:
469
- state_dict[pair[0]] = state_dict[pair[1]]
470
-
471
- return state_dict
472
-
473
-
474
- def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
475
- """
476
- Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
477
- ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
478
- via a model hub.
479
-
480
- Args:
481
- - ``checkpoint_dir``: path to the desired checkpoint folder
482
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
483
- - ``exclude_frozen_parameters``: exclude frozen parameters
484
-
485
- Returns:
486
- - pytorch ``state_dict``
487
-
488
- Note: this approach may not work if your application doesn't have sufficient free CPU memory and
489
- you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
490
- the checkpoint.
491
-
492
- A typical usage might be ::
493
-
494
- from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
495
- # do the training and checkpoint saving
496
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
497
- model = model.cpu() # move to cpu
498
- model.load_state_dict(state_dict)
499
- # submit to model hub or save the model to share with others
500
-
501
- In this example the ``model`` will no longer be usable in the deepspeed context of the same
502
- application. i.e. you will need to re-initialize the deepspeed engine, since
503
- ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
504
-
505
- If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
506
-
507
- """
508
- if tag is None:
509
- latest_path = os.path.join(checkpoint_dir, 'latest')
510
- if os.path.isfile(latest_path):
511
- with open(latest_path, 'r') as fd:
512
- tag = fd.read().strip()
513
- else:
514
- raise ValueError(f"Unable to find 'latest' file at {latest_path}")
515
-
516
- ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
517
-
518
- if not os.path.isdir(ds_checkpoint_dir):
519
- raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
520
-
521
- return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
522
-
523
-
524
- def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir, output_file, tag=None, exclude_frozen_parameters=False):
525
- """
526
- Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
527
- loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
528
-
529
- Args:
530
- - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
531
- - ``output_file``: path to the pytorch fp32 state_dict output file (e.g. path/pytorch_model.bin)
532
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
533
- - ``exclude_frozen_parameters``: exclude frozen parameters
534
- """
535
-
536
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
537
- print(f"Saving fp32 state dict to {output_file}")
538
- torch.save(state_dict, output_file)
539
-
540
-
541
- def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
542
- """
543
- 1. Put the provided model to cpu
544
- 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
545
- 3. Load it into the provided model
546
-
547
- Args:
548
- - ``model``: the model object to update
549
- - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
550
- - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
551
-
552
- Returns:
553
- - ``model`: modified model
554
-
555
- Make sure you have plenty of CPU memory available before you call this function. If you don't
556
- have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
557
- conveniently placed for you in the checkpoint folder.
558
-
559
- A typical usage might be ::
560
-
561
- from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
562
- model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
563
- # submit to model hub or save the model to share with others
564
-
565
- Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
566
- of the same application. i.e. you will need to re-initialize the deepspeed engine, since
567
- ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
568
-
569
- """
570
- logger.info(f"Extracting fp32 weights")
571
- state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
572
-
573
- logger.info(f"Overwriting model with fp32 weights")
574
- model = model.cpu()
575
- model.load_state_dict(state_dict, strict=False)
576
-
577
- return model
578
-
579
-
580
- if __name__ == "__main__":
581
-
582
- parser = argparse.ArgumentParser()
583
- parser.add_argument("checkpoint_dir",
584
- type=str,
585
- help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
586
- parser.add_argument(
587
- "output_file",
588
- type=str,
589
- help="path to the pytorch fp32 state_dict output file (e.g. path/checkpoint-12/pytorch_model.bin)")
590
- parser.add_argument("-t",
591
- "--tag",
592
- type=str,
593
- default=None,
594
- help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
595
- parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
596
- parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
597
- args = parser.parse_args()
598
-
599
- debug = args.debug
600
-
601
- convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
602
- args.output_file,
603
- tag=args.tag,
604
- exclude_frozen_parameters=args.exclude_frozen_parameters)