AssistantsLab
/

SmolLM2-360M-humanized

@@ -43,7 +43,7 @@ pip install transformers
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-checkpoint = "AssistantsLab/SmolLM2-135M-humanized"
 device = "cuda" # for GPU usage or "cpu" for CPU usage
 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
@@ -62,7 +62,7 @@ print(tokenizer.decode(outputs[0]))
 You can also use the TRL CLI to chat with the model from the terminal:
 ```bash
 pip install trl
-trl chat --model_name_or_path AssistantsLab/SmolLM2-135M-humanized --device cpu
 ```
 ## Evaluation
@@ -73,16 +73,16 @@ In this section, we report the evaluation results of SmolLM2. All evaluations ar
 | Metric                       | SmolLM2-135M-Instruct | SmolLM2-135M-Humanized | Difference |
 |:-----------------------------|:---------------------:|:----------------------:|:----------:|
-| MMLU                         | **24.0**              | 23.0                   | -1.1       |
-| ARC (Easy)                   | **56.4**              | 55.4                   | -1.1       |
-| ARC (Challenge)              | **31.7**              | 25.7                   | -6.0       |
-| HellaSwag                    | **56.9**              | 42.5                   | -14.3      |
-| PIQA                         | **71.6**              | 67.1                   | -4.3       |
-| WinoGrande                   | **54.8**              | 51.4                   | -3.4       |
-| TriviaQA                     | 0.2                   | **0.3**                | +0.1       |
-| GSM8K                        | 0.3                   | **0.7**                | +0.4       |
-| OpenBookQA                   | **36.2**              | 33.0                   | -3.6       |
-| QuAC (F1)                    | **23.0**              | 13.3                   | -9.2       |
 ## Limitations

 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
+checkpoint = "AssistantsLab/SmolLM2-360M-humanized"
 device = "cuda" # for GPU usage or "cpu" for CPU usage
 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
 You can also use the TRL CLI to chat with the model from the terminal:
 ```bash
 pip install trl
+trl chat --model_name_or_path AssistantsLab/SmolLM2-360M-humanized --device cpu
 ```
 ## Evaluation
 | Metric                       | SmolLM2-135M-Instruct | SmolLM2-135M-Humanized | Difference |
 |:-----------------------------|:---------------------:|:----------------------:|:----------:|
+| MMLU                         | 24.0                  | **25.4**               | +1.4       |
+| ARC (Easy)                   | **56.4**              | 54.8                   | -1.6       |
+| ARC (Challenge)              | **31.7**              | 28.8                   | -2.9       |
+| HellaSwag                    | **56.9**              | 54.9                   | -2.0       |
+| PIQA                         | **71.6**              | 70.6                   | -1.0       |
+| WinoGrande                   | 54.8                  | **54.9**               | +0.1       |
+| TriviaQA                     | **0.2**               | **0.2**                | +0.0       |
+| GSM8K                        | **0.3**               | 0.1                    | -0.2       |
+| OpenBookQA                   | **36.2**              | 35.8                   | -0.4       |
+| QuAC (F1)                    | 23.0                  | **23.7**               | +0.7       |
 ## Limitations