---
license: mit
language:
- en
- zh
---

# Model Card for sparsing-law-0.1b-relu

- **Paper:** [paper](https://arxiv.org/pdf/2411.02335)
- **Repository and demo code:** [github](https://github.com/thunlp/SparsingLaw)

This model is ReLU-activated and contains approximately 0.1 billion non-embedding parameters.

The model was trained from scratch using the pre-training dataset described in our paper, with the WSD (Warmup-Stable-Decay) learning rate scheduler. It represents the final checkpoint of the stable stage in WSD, meaning it has not undergone the decay stage.