---
license: apache-2.0
---

## Overview
LightLM is a series of 3 language models trained on open-access data (Cosmopedia v2). We present three configurations (one with Mixture-of-Experts and two without) that aim to optimize parameter distribution between Attention and Feed-Forward layers. Despite a relatively modest training corpus of ~28B tokens, these models approach or surpass performance of other models in their parameter range (e.g., MobileLLM, GPT-neo-125M).


1. **Model 1 ([Model Attn](https://huggingface.co/Virg1n/LightLM/tree/main/Model%20Attn))**
   - **Layers**: 34  
   - **Attention dim**: 832  
   - **FFN dim**: 556  
   - **Context length**: 1536  

2. **Model 2 ([Model FFN](https://huggingface.co/Virg1n/LightLM/tree/main/Model%20FFN))**
   - **Layers**: 32  
   - **Attention dim**: 512  
   - **FFN dim**: 512 × 4 = 2048  
   - **Context length**: 1536  

3. **Model 3 ([Model MoE 2+1](https://huggingface.co/Virg1n/LightLM/tree/main/Model%20MoE%202%2B1))**  
   - **Layers**: 32  
   - **Attention dim**: 384 (experimental setting)  
   - **FFN**: 2 routed experts + 1 shared expert  
     - Each expert has 512 × 2 = 1024 hidden units  
     - 100% of parameters are active; router assigns expert weights per token  
   - **Context length**: 1024  



## Results
| **Model**            | **#Params** | **ARC-c** | **WinoGrande** |
|----------------------|-------------|-----------|----------------|
| GPT-neo-125M         | 125M        | 24.8      | 50.7           |
| Pythia-160M          | 162M        | 25.3      | 50.9           |
| RWKV-169M            | 169M        | 25.3      | 51.5           |
| MobileLLM-125M       | 125M        | 27.1      | 53.1           |
| LightLM (Attn)       | 146M        | 25.1      | 52.0           |
| LightLM (FFN)        | 146M        | 27.2      | 47.5           |
| LightLM (MoE)        | 144M        | 26.3      | 52.8           |


**Example Output**  
Prompt: `"Hello, I am a language model,"`  
```
Hello, I am a language model, and I can help you learn more about the language you are interested in. 
Let's start with the basics.
```
```
Hello, I am a language model, and I can help you learn some new words and phrases. Maybe you could try 
saying "hello" in English first, then move on to Spanish, ...
```

[🔗 View on GitHub](https://github.com/virg1n/LightLM)