Update README.md

README.md

@@ -72,16 +72,18 @@ print(tokenizer.decode(output))
 - **Model type:** Transformer-based Language Model
 - **Total seen tokens:** 2.1T tokens
 
-|Params|Layers|Hidden size|Heads|Context length|Embedding parameters|Non-embedding parameters|
-|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
-|150M|12|512|8|4096|101,874,688|50,344,448|
-|440M|16|1024|8|4096|203,749,376|243,303,424|
-|980M|20|1536|8|4096|305,624,064|684,258,816|
-|1.8b|24|2048|16|4096|407,498,752|1,459,718,144|
-|3.7b|28|3072|24|4096|611,248,128|3,171,068,928|
-|7.2b|32|4096|32|4096|814,997,504|6,476,271,616|
-|13b|40|5120|40|4096|1,018,746,880|12,688,184,320|
-|172b|96|12288|96|4096|2,444,992,512|169,947,181,056|
+|Params|Layers|Hidden size|Heads|Routed Experts|Activated Experts|Context length|Embedding parameters|Non-embedding parameters|Activated parameters|Total parameters|
+|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+|150M|12|512|8|-|-|4096|101,874,688|50,344,448|152,219,136|152,219,136|
+|440M|16|1024|8|-|-|4096|203,749,376|243,303,424|447,052,800|447,052,800|
+|980M|20|1536|8|-|-|4096|305,624,064|684,258,816|989,882,880|989,882,880|
+|1.8b|24|2048|16|-|-|4096|407,498,752|1,459,718,144|1,867,216,896|1,867,216,896|
+|8x1.8b|24|2048|16|8|2|4096|407,498,752|8,858,863,616|2,924,279,808|9,266,362,368|9,266,362,368|
+|3.7b|28|3072|24|-|-|4096|611,248,128|3,171,068,928|3,782,317,056|3,782,317,056|
+|7.2b|32|4096|32|-|-|4096|814,997,504|6,476,271,616|7,291,269,120|7,291,269,120|
+|13b|40|5120|40|-|-|4096|1,018,746,880|12,688,184,320|13,706,931,200|13,706,931,200|
+|8x13b|40|5120|40|8|2|4096|1,018,746,880|72,144,081,920|22,200,806,400|73,162,828,800|
+|172b|96|12288|96|-|-|4096|2,444,992,512|169,947,181,056|172,392,173,568|172,392,173,568|
 
 ## Tokenizer