File size: 2,778 Bytes
249b7d4 976a963 249b7d4 976a963 da27324 249b7d4 976a963 249b7d4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
---
license: apache-2.0
datasets:
- wikimedia/wikipedia
- custom
language:
- en
pipeline_tag: text-generation
---
## Description
This is "Lemnos" , a new Instruction Tuned model based on the Llama 2 model architecture.
It was trained on general wikipedia corpus and then finetuned on a custom instruction dataset.
It is only for use as an experimental version prior launching a new one which also supports Greek.
## Usage:
Prerequisites packages:
- transformers
- accelerate
- bitsandbytes-0.43.1
Minimum Environment: T4 GPU (The free of charge Google Colab T4, should run fine)
or just run all this Colab (make sure you select T4 GPU):
https://colab.research.google.com/drive/1lp-JygPxsaQp-NdB7Mh_uVVYeIIXcAlt?usp=sharing
Notice: Since it is a 7B parameter model, in FP32 and it takes some time to load all the safetensors.
An alternative, 4-bit quantized will be uploaded soon.
```python
# Upgrade in case bitsandbytes already installed
pip install transformers accelerate bitsandbytes -U
# or from Colab
!pip install transformers accelerate bitsandbytes -U
```
```python
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch
# Specify the model hub
hub_model = 'gsar78/Lemnos_it_en_v2'
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(hub_model, trust_remote_code=True)
# Configure the BitsAndBytesConfig for 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=False
)
# Load the model with the specified configuration
model = AutoModelForCausalLM.from_pretrained(
hub_model,
quantization_config=bnb_config,
trust_remote_code=True,
device_map="auto"
)
# Function to generate text based on a prompt using the Alpaca format
def generate_text(prompt, max_length=512):
# Format the prompt according to the Alpaca format
alpaca_prompt = f"### Instruction:\n{prompt}\n\n### Response:\n"
# Tokenize the input prompt
inputs = tokenizer(alpaca_prompt, return_tensors="pt").to(model.device)
# Generate text using the model
outputs = model.generate(
input_ids=inputs['input_ids'],
max_length=max_length,
num_return_sequences=1,
pad_token_id=tokenizer.eos_token_id
)
# Decode the generated tokens to text
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Remove the prompt part from the output to get only the response
response = generated_text[len(alpaca_prompt):]
return response
# Example question
prompt = "What are the three basic colors?"
generated_text = generate_text(prompt)
print(generated_text)
# Output:
# Red, blue, and yellow.
``` |