File size: 3,895 Bytes
67efbea 508b89b 67efbea |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
---
license: other
license_name: apple-sample-code-license
license_link: LICENSE
---
# OpenELM
*Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari*
We introduce **OpenELM**, a family of **Open**-source **E**fficient **L**anguage **M**odels. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters.
Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens.
See the list below for the details of each model:
- [OpenELM-270M](https://huggingface.co/apple/OpenELM-270M)
- [OpenELM-450M](https://huggingface.co/apple/OpenELM-450M)
- [OpenELM-1_1B](https://huggingface.co/apple/OpenELM-1_1B)
- [OpenELM-3B](https://huggingface.co/apple/OpenELM-3B)
- [OpenELM-270M-Instruct](https://huggingface.co/apple/OpenELM-270M-Instruct)
- [OpenELM-450M-Instruct](https://huggingface.co/apple/OpenELM-450M-Instruct)
- [OpenELM-1_1B-Instruct](https://huggingface.co/apple/OpenELM-1_1B-Instruct)
- [OpenELM-3B-Instruct](https://huggingface.co/apple/OpenELM-3B-Instruct)
```python
from transformers import AutoModelForCausalLM
openelm_270m = AutoModelForCausalLM.from_pretrained("apple/OpenELM-270M", trust_remote_code=True)
openelm_450m = AutoModelForCausalLM.from_pretrained("apple/OpenELM-450M", trust_remote_code=True)
openelm_1b = AutoModelForCausalLM.from_pretrained("apple/OpenELM-1_1B", trust_remote_code=True)
openelm_3b = AutoModelForCausalLM.from_pretrained("apple/OpenELM-3B", trust_remote_code=True)
openelm_270m_instruct = AutoModelForCausalLM.from_pretrained("apple/OpenELM-270M-Instruct", trust_remote_code=True)
openelm_450m_instruct = AutoModelForCausalLM.from_pretrained("apple/OpenELM-450M-Instruct", trust_remote_code=True)
openelm_1b_instruct = AutoModelForCausalLM.from_pretrained("apple/OpenELM-1_1B-Instruct", trust_remote_code=True)
openelm_3b_instruct = AutoModelForCausalLM.from_pretrained("apple/OpenELM-3B-Instruct", trust_remote_code=True)
```
## Example Usage
Below we provide an example of loading the model via [HuggingFace Hub](https://huggingface.co/docs/hub/) as:
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
# obtain access to "meta-llama/Llama-2-7b-hf", then see https://huggingface.co/docs/hub/security-tokens to get a token
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", token="hf_xxxx")
model_path = "apple/OpenELM-450M"
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
model = model.cuda().eval()
prompt = "Once upon a time there was"
tokenized_prompt = tokenizer(prompt)
prompt_tensor = torch.tensor(tokenized_prompt["input_ids"], device="cuda").unsqueeze(0)
output_ids = model.generate(prompt_tensor, max_new_tokens=256, repetition_penalty=1.2, pad_token_id=0)
output_ids = output_ids[0].tolist()
output_text = tokenizer.decode(output_ids, skip_special_tokens=True)
print(f'{model_path=}, {prompt=}\n')
print(output_text)
# below is the output:
"""
model_path='apple/OpenELM-450M', prompt='Once upon a time there was'
Once upon a time there was a little girl who lived in the woods. She had a big heart and she loved to play with her friends. One day, she decided to go for a walk in the woods. As she walked, she saw a beautiful tree. It was so tall that it looked like a mountain. The tree was covered with leaves and flowers.
The little girl thought that this tree was very pretty. She wanted to climb up to the tree and see what was inside. So, she went up to the tree and climbed up to the top. She was very excited when she saw that the tree was full of beautiful flowers. She also
"""
```
|