research/norwegian_llm_research.md

# Norwegian LLM and Embedding Models Research

## Open-Source LLMs with Norwegian Language Support

### 1. NorMistral-7b-scratch
- **Description**: A large Norwegian language model pretrained from scratch on 260 billion subword tokens (using six repetitions of open Norwegian texts).
- **Architecture**: Based on Mistral architecture with 7 billion parameters
- **Context Length**: 2k tokens
- **Performance**: 
  - Perplexity on NCC validation set: 7.43
  - Good performance on reading comprehension, sentiment analysis, and machine translation tasks
- **License**: Apache-2.0
- **Hugging Face**: https://huggingface.co/norallm/normistral-7b-scratch
- **Notes**: Part of the NORA.LLM family developed by the Language Technology Group at the University of Oslo

### 2. Viking 7B
- **Description**: The first multilingual large language model for all Nordic languages (including Norwegian)
- **Architecture**: Similar to Llama 2, with flash attention, rotary embeddings, grouped query attention
- **Context Length**: 4k tokens
- **Performance**: Best-in-class performance in all Nordic languages without compromising English performance
- **License**: Apache 2.0
- **Notes**: 
  - Developed by Silo AI and University of Turku's research group TurkuNLP
  - Also available in larger sizes (13B and 33B parameters)
  - Trained on 2 trillion tokens including Danish, English, Finnish, Icelandic, Norwegian, Swedish and programming languages

### 3. NorskGPT
- **Description**: A Norwegian large language model made for Norwegian society
- **Versions**:
  - NorskGPT-Mistral: 7B dense transformer with 8K context window, based on Mistral 7B
  - NorskGPT-LLAMA2: 7b and 13b parameter model with 4K context length, based on LLAMA2
- **License**: cc-by-nc-sa-4.0 (non-commercial)
- **Website**: https://www.norskgpt.com/norskgpt-llm

## Embedding Models for Norwegian

### 1. NbAiLab/nb-sbert-base
- **Description**: A SentenceTransformers model trained on a machine translated version of the MNLI dataset
- **Architecture**: Based on nb-bert-base
- **Vector Dimensions**: 768
- **Performance**: 
  - Cosine Similarity: Pearson 0.8275, Spearman 0.8245
- **License**: apache-2.0
- **Hugging Face**: https://huggingface.co/NbAiLab/nb-sbert-base
- **Use Cases**:
  - Sentence similarity
  - Semantic search
  - Few-shot classification (with SetFit)
  - Keyword extraction (with KeyBERT)
  - Topic modeling (with BERTopic)
- **Notes**: Works well with both Norwegian and English, making it ideal for bilingual applications

### 2. FFI/SimCSE-NB-BERT-large
- **Description**: A Norwegian sentence embedding model trained using the SimCSE methodology
- **Hugging Face**: https://huggingface.co/FFI/SimCSE-NB-BERT-large

## Vector Database Options for Hugging Face RAG Integration

### 1. Milvus
- **Integration**: Well-documented integration with Hugging Face for RAG pipelines
- **Reference**: https://huggingface.co/learn/cookbook/en/rag_with_hf_and_milvus

### 2. MongoDB
- **Integration**: Can be used with Hugging Face models for RAG systems
- **Reference**: https://huggingface.co/learn/cookbook/en/rag_with_hugging_face_gemma_mongodb

### 3. MyScale
- **Integration**: Supports building RAG applications with Hugging Face embedding models
- **Reference**: https://medium.com/@myscale/building-a-rag-application-in-10-min-with-claude-3-and-hugging-face-10caea4ea293

### 4. FAISS (Facebook AI Similarity Search)
- **Integration**: Lightweight vector database that works well with Hugging Face
- **Notes**: Can be used with `autofaiss` for quick experimentation

## Hugging Face RAG Implementation Options

1. **Transformers Library**: Provides access to pre-trained models
2. **Sentence Transformers**: For text embeddings
3. **Datasets**: For managing and processing data
4. **LangChain Integration**: For advanced RAG pipelines
5. **Spaces**: For deploying and sharing the application