asiansoul/YACHT-Llama-3-Ko-8B-GGUF

YACHT-Llama-3-Ko-8B

🎵 [JayLee LLMs Signature Tag] : ✍️ "I need a Jay Jay chat boy" 🎵

✨ Navigating the High Seas of Data: Crafting the Ultimate Yacht Insights with Merged LLMs ✨

✨ Aren’t you sometimes tired of just doing LLM & RAG & Normal Chat app? I'll show you a cool app soon integrating this my merged one(Tuned car). It wouldn't be fun if we only developed cars, so life is ultimately about driving cars and socializing with people. ✨

🧨 When using the Merge model for commercial purposes, a lot of care is needed. A mix of many models can be good, but it can also pose many risks. 🧨

🏟️ Merged Model Series Yacht Features

Welcome to the merged model series yacht! This provides an overview of the powerful features and functionalities that this series brings together, akin to a sleek, modern yacht sailing across the digital ocean.

1. Function Calling & JSON Outputs

• Offers precise function calling and structured JSON outputs via specialized tokens like <tools>, <tool_call>, and <tool_response>. Streamlines system communication for developers.

2. Conversational Interaction

• Avoids excessive "SYSTEM MESSAGE" chatter while delivering seamless, friendly dialogue.
• Specializes in answering questions with precision, handling arithmetic and tabular data effortlessly.

3. Expanded Context Length

• Extends the context length to 256k tokens using PoSE, offering a broader field of data analysis.

4. Multilingual Capabilities

• Transfers instruction-following from English to Korean for reliable interaction across languages.

5. Optimized Dialogue & Safety

• Aligns with human preferences using fine-tuning (SFT) and reinforcement learning (RLHF), ensuring helpful and safe dialogues.

6. Precision Merging

• Merges foundational and preview models for Korean language through task arithmetic, providing seamless integration.

7. Specialized Biomedical Knowledge

• Specializes in biomedical tasks with accurate responses for healthcare professionals and researchers.

8. Novel Training & Collaboration

• Combines ORPO method and dolphin preference datasets for high-quality conversation and collaboration.

The merged model series yacht offers unparalleled functionality, drawing together a fleet of specialized models. Whether you need precise function calling, multilingual capabilities, or conversational AI, this yacht has every deck optimized to navigate the digital ocean with style and precision.

👘 Merge Method

This model was merged using the DARE TIES merge method using NousResearch/Meta-Llama-3-8B as a base.

🩱 Models Merged

The following models were included in the merge:

• NousResearch/Hermes-2-Pro-Llama-3-8B
• cognitivecomputations/dolphin-2.9-llama3-8b
• winglian/llama-3-8b-256k-PoSE
• maum-ai/Llama-3-MAAL-8B-Instruct-v0.1
• asiansoul/Llama-3-Open-Ko-Linear-8B
• NousResearch/Meta-Llama-3-8B-Instruct
• nvidia/Llama3-ChatQA-1.5-8B
• Danielbrdz/Barcenas-Llama3-8b-ORPO
• aaditya/Llama3-OpenBioLLM-8B

💃 ModelFile

The parameters below depend on the performance of your Ollma-based computer. Therefore, the settings below do not necessarily mean that the performance will be as good.

ollama create yacht -f ./Modelfile_Q5_K_M

FROM yacht-llama-3-ko-8b-Q5_K_M.gguf
TEMPLATE """
{{- if .System }}
system
<s>{{ .System }}</s>
{{- end }}
user
<s>Human:
{{ .Prompt }}</s>
assistant
<s>Assistant:
"""

SYSTEM """
친절한 챗봇으로서 상대방의 요청에 최대한 자세하고 친절하게 답하자. 모든 대답은 한국어(Korean)으로 대답해줘.
"""

PARAMETER temperature 0.7
PARAMETER num_predict 3000
PARAMETER num_ctx 250000
PARAMETER stop "<s>"
PARAMETER stop "</s>"

🪭 Configuration

Computer System

Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 64 GB
      System Firmware Version: 10151.101.3

The following YAML configuration was used to produce this model:

models:
  - model: NousResearch/Meta-Llama-3-8B
    # Base model providing a general foundation without specific parameters

  - model: NousResearch/Meta-Llama-3-8B-Instruct
    parameters:
      density: 0.60  
      weight: 0.25  
  
  - model: winglian/llama-3-8b-256k-PoSE
    parameters:
      density: 0.55  
      weight: 0.15  
  
  - model: nvidia/Llama3-ChatQA-1.5-8B
    parameters:
      density: 0.55  
      weight: 0.1
  
  - model: asiansoul/Llama-3-Open-Ko-Linear-8B
    parameters:
      density: 0.55  
      weight: 0.2  

  - model: maum-ai/Llama-3-MAAL-8B-Instruct-v0.1
    parameters:
      density: 0.55  
      weight: 0.1 

  - model: NousResearch/Hermes-2-Pro-Llama-3-8B
    parameters:
      density: 0.55  
      weight: 0.1  

  - model: cognitivecomputations/dolphin-2.9-llama3-8b
    parameters:
      density: 0.55  
      weight: 0.05  

  - model: Danielbrdz/Barcenas-Llama3-8b-ORPO
    parameters:
      density: 0.55  
      weight: 0.05 

  - model: aaditya/Llama3-OpenBioLLM-8B
    parameters:
      density: 0.55  
      weight: 0.1 

merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
parameters:
  int8_mask: true
dtype: bfloat16

⌨️ Codes

Code that serves as a summary

• TXT, PDF, WiKi Summary Code supporting both Korean, English By BrianHan(my follower) -> Because several models are mixed, if you attach a large file and ask a question in Korean and request a summary answer in Korean, sometimes the answer is given in English only for the large ctx cuz based on english maybe(for the several model test). In that case, I asked them to translate it into Korean using Google Translator. Even though the answer was sent in English, I confirmed that it was interpreted accurately and sent.

(.venv) jaylee@lees-MacBook-Pro-2 tmp % python han.py -m yacht:latest -s steve.txt -l en
Model: yacht:latest, Langguage: en
Source : steve.txt
Processing input...
Reading text file...
Summarizing content...using ollama model : yacht:latest 

import sys
import os
import requests
from bs4 import BeautifulSoup
import PyPDF2
from langchain_community.chat_models import ChatOllama
from langchain.schema import AIMessage, HumanMessage, SystemMessage
from googletrans import Translator
from langchain.chains.summarize import load_summarize_chain
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.prompts  import PromptTemplate
from langchain.docstore.document import Document
import argparse
import logging

def clean_output(text):
    text = text.replace("</s>", "").strip()
    return text

def translate_text(text, src_lang, dest_lang):
    """Translates text from source language to destination language using Google Translate."""
    if src_lang == dest_lang:
        return text
    translator = Translator()
    try:
        translation = translator.translate(text, src=src_lang, dest=dest_lang)
        return translation.text
    except Exception as e:
        logging.error(f"Translation failed: {e}")
        return text

def detect_language(text):
    """Detects the language of the given text."""
    translator = Translator()
    try:
        detected = translator.detect(text)
        return detected.lang
    except Exception as e:
        logging.error(f"Language detection failed: {e}")
        return None

def refine_summary(data,target_lang,ollama_model):
    if target_lang == 'ko':
            prompt_template = """
                다음 텍스트에 대한 전문적인 요약을 제공하여 주세요. 요약은 한국어의 언어적 뉘앙스에 맞게 최고 수준의 명확성과 세부 사항을 준수해애 합니다
                Text: `{text}`
                CONCISE SUMMARY : """
            refine_template= """
                최종 요약을 제공하여 주세요.
                중간 요약을 제공 다음과 같이 제공합니다 : {existing_answer}.
                최종 요약은 한국어의 언어적 뉘앙스에 맞게 최고 수정의 명확성과 세부 사항을 준수해야 합니다.
                -------------
                {text}
                -------------
                최종 요약은 도입부가 있어야 합니다. 도입부는 전체 주제를 제공하여야 합니다.
                본문은 BULLET POINT로 시작하여야 합니다. 마지막은 결론부가 있어야 합니다. 결론부분은 최종 요약의 결론이어야 합니다.  """

    else:  # default to English if not Korean
            prompt_template = """
                Write a concise summary of the following extracting the key information:
                Text: `{text}`
                CONCISE SUMMARY : """
            refine_template= """
                Your job is to produce a final summary.
                I have provided an existing summary up to a certain point : {existing_answer}.
                Please refine eht existing summary with some more context blow.
                -------------
                {text}
                -------------
                Start the final summary with an INTRODUCTION PARAGRAPH that gives an overview of the the topic FOLLOWED 
                by BULLET POINTS if possible AND end the summary with CONCLUSION PHRASE.  """

    docs = [Document(page_content=data)]


    initial_prompt = PromptTemplate(template=prompt_template, input_variables=['text'])
    refine_prompt = PromptTemplate(
        template=refine_template,
        input_variables=['existing_answer', 'text']
    )
    llm = ChatOllama(model=ollama_model)
    chain = load_summarize_chain(
        llm=llm,
        chain_type='refine',
        question_prompt=initial_prompt,
        refine_prompt=refine_prompt,
        return_intermediate_steps=False
    )

    output_summary =  chain.invoke(docs)

    output_summary['output_text']
    cleaned_content = clean_output(output_summary['output_text'])
    # print(cleaned_content)
    content_lang = detect_language(cleaned_content)
    print(f"Current content language: {content_lang}, Target language to be translated to: {target_lang}")
    if content_lang != target_lang:
        return translate_text(cleaned_content, content_lang, target_lang)

    return cleaned_content


    



def invoke_model(text,target_lang,ollama_model):

    if target_lang == 'ko':
            messages = [
                SystemMessage(content='문서의 핵심 요약을 상세하게 제공해 주실 전문가로서, 다음 문서를 요약해 주세요.'),
                HumanMessage(content=f'다음 텍스트에 대한 전문적 요약을 제공해 주세요. 요약은 한국어의 언어적 뉘앙스에 맞게 최고 수준의 명확성과 세부 사항을 준수해야 합니다:\n\nTEXT: {text}')
            ]
    else:  # default to English if not Korean
            messages = [
                SystemMessage(content='As an adept summarizer, your expertise is required to condense the following document into its essential points in detail.'),
                HumanMessage(content=f'Kindly provide an expert summary of the text below, adhering to the highest standards of clarity and detail. Ensure the response is tailored to the linguistic nuances of English:\n\nTEXT: {text}')
            ]
    
    try:
        llm = ChatOllama(model=ollama_model)
        summary_output = llm.invoke(messages)
        if isinstance(summary_output, AIMessage):
            cleaned_content = clean_output(summary_output.content)
            # print(cleaned_content)
            content_lang = detect_language(cleaned_content)
            print(f"Current content language: {content_lang}, Target language to be translated to: {target_lang}")
            if content_lang != target_lang:
                return translate_text(cleaned_content, content_lang, target_lang)

            return cleaned_content
        else:
            return "Unexpected data type for model output."
    except Exception as e:
        print(f"An error occurred while processing the model output: {str(e)}")
        return None

def fetch_text_from_url(url):
    try:
        response = requests.get(url)
        response.raise_for_status()
        soup = BeautifulSoup(response.text, 'html.parser')
        main_content = soup.select_one('#mw-content-text, #bodyContent, .content')
        if not main_content:
            logging.error("No content found in the expected sections.")
            return None
        text_content = ' '.join(p.get_text() for p in main_content.find_all(['p', 'li'], string=True))
        return text_content
    except requests.RequestException as e:
        print(f"Failed to fetch data from URL: {str(e)}")
        return None

def read_text_file(file_path):
    with open(file_path, "r", encoding="utf-8") as file:
        return file.read()

def read_pdf(file_path):

    '''
    with open(file_path, "rb") as file:
        reader = PyPDF2.PdfReader(file)
        text_content = ""
        for page in reader.pages:
            extracted_text = page.extract_text()
            if extracted_text:
                text_content += extracted_text + "\n"
        return text_content
    '''
    try:
       with open(file_path, "rb") as file:
            reader = PyPDF2.PdfReader(file)
            return ' '.join(page.extract_text() for page in reader.pages if page.extract_text())
    except Exception as e:
            logging.error(f"Error reading PDF file: {e}")
            return None

def summarize_content(source,language,model):
    print("Processing input...")
    text_content = None
    if source.startswith(('http://', 'https://')):
        print("Fetching content from URL...")
        text_content = fetch_text_from_url(source)
    elif os.path.isfile(source):
        _, file_extension = os.path.splitext(source)
        if file_extension.lower() == '.pdf':
            print("Reading PDF...")
            text_content = read_pdf(source)
        elif file_extension.lower() in ['.txt', '.text']:
            print("Reading text file...")
            text_content = read_text_file(source)
        else:
            print("Unsupported file type")
            return
    else:
        print("Unsupported file type")
        return
    
    if text_content:
        print(f"Summarizing content...using ollama model : {model} ")
        summary = refine_summary(text_content, language,model)

        print("\n--- Summary of the document ---\n")
        print(summary)
    else:
        print("No text found or unable to extract text from source.")


if __name__ == '__main__':
    defSrc ='https://en.wikipedia.org/wiki/Britney_Spears'
    defModel='llama3:latest'
    defLang = 'en'

    parser = argparse.ArgumentParser()

    parser.add_argument('-s',dest='source',default = defSrc,help='text file or URL');
    parser.add_argument('-m',dest='model',default = defModel,help='ollama Model');
    parser.add_argument('-l',dest='language',default = defLang,help='en for English, kr for Korean');
    arg= parser.parse_args()
    print(f"Model: {arg.model}, Langguage: {arg.language}")
    print(f"Source : {arg.source}")

    summarize_content(arg.source,arg.language,arg.model)
    '''
    if len(sys.argv) < 2:
        print("Usage: python script.py <file_path_or_url>")
    else:
        source = sys.argv[1]
        summarize_content(source)
    '''

🚴 Test Result(Streamlit + Ollama Server)

• Normal Question

• Tech Question

🚵‍♂️ Test Result(Summary Code)