README.md

metadata

library_name: transformers
tags: []

Llama-3-SURPASSONE-JP-8B

Model Description

Llama-3-SURPASSONE-JP-8B is a large language model trained by SURPASSONE, Inc. Based on meta-llama/Meta-Llama-3-8B-Instruct, it has been enhanced for Japanese usage through additional pre-training and instruction tuning. (Built with Meta Llama3)

For more details, please refer to our blog post.

Usage

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer

bnb_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_use_double_quant=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_compute_dtype=torch.bfloat16,
)

model_id = "surpassone/Llama-3-SURPASSONE-JP-8B"
            
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
   model_id,
   device_map="auto",
   quantization_config=None, # Use bnb_config, if need to use 4 bit quantization else None
)
if tokenizer.pad_token is None:
   tokenizer.pad_token = tokenizer.eos_token

model.eval()
alpaca_prompt = """以下は、タスクを説明する指示と、さらに詳しいコンテキストを提供する入力を組み合わせたものです。要求を適切に完了する応答を記述してください。

### 説明書:
{}

### 入力:
{}

### 応答:
{}"""

EOS_TOKEN = "<|endoftext|>"  # Define the EOS token, adjust according to your tokenizer

inputs = tokenizer(
[
   alpaca_prompt.format(
       "次のトピックに関する複数選択問題を生成します。", # instruction
       "介護：体の仕組み", # input
       "", # output - leave this blank for generation!
   )
], return_tensors = "pt").to("cuda")

from transformers import TextStreamer
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 1028)