Update README.md
Browse files
README.md
CHANGED
|
@@ -1,4 +1,6 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
| 2 |
router is roughly 70M params
|
| 3 |
|
| 4 |
no loss chart for this
|
|
|
|
| 1 |
+
# !! "moe" - routed inference between 3 different models without any tying
|
| 2 |
+
|
| 3 |
+
experimental MoE with 3 experts totalling 480m~ params
|
| 4 |
router is roughly 70M params
|
| 5 |
|
| 6 |
no loss chart for this
|