alugowski commited on
Commit
9fb9253
·
verified ·
1 Parent(s): c8f4336

Update config.json

Browse files

Set key value heads.
Previously was explicitly null, as the logic is per block. In vLLM code:
```
num_kv_heads = (config.num_attention_heads //
block_config.attention.n_heads_in_group)
```

Setting default number here: 64/8 = 8

Files changed (1) hide show
  1. config.json +1 -1
config.json CHANGED
@@ -1461,7 +1461,7 @@
1461
  "model_type": "nemotron-nas",
1462
  "num_attention_heads": 64,
1463
  "num_hidden_layers": 80,
1464
- "num_key_value_heads": null,
1465
  "pretraining_tp": 1,
1466
  "quantization_config": {
1467
  "config_groups": {
 
1461
  "model_type": "nemotron-nas",
1462
  "num_attention_heads": 64,
1463
  "num_hidden_layers": 80,
1464
+ "num_key_value_heads": 8,
1465
  "pretraining_tp": 1,
1466
  "quantization_config": {
1467
  "config_groups": {